Sep 17 17:56:51.800219 ip-10-0-130-28 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 17:56:52.183372 ip-10-0-130-28 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:56:52.183372 ip-10-0-130-28 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 17:56:52.183372 ip-10-0-130-28 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:56:52.183372 ip-10-0-130-28 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 17:56:52.183372 ip-10-0-130-28 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:56:52.184277 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.184196    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 17:56:52.189237 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189214    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:56:52.189237 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189234    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:56:52.189237 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189239    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:56:52.189237 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189242    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:56:52.189237 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189245    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:56:52.189432 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189248    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:56:52.189432 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189251    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:56:52.189432 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189254    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:56:52.189432 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189257    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:56:52.189432 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189260    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:56:52.189432 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189262    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:56:52.189432 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189265    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:56:52.189432 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189267    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:56:52.189432 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189270    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:56:52.189432 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189273    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:56:52.189432 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189275    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:56:52.189432 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189278    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:56:52.189432 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189280    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:56:52.189432 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189283    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:56:52.189432 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189285    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:56:52.189432 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189288    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:56:52.189432 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189290    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:56:52.189432 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189293    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:56:52.189432 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189296    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:56:52.189432 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189298    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:56:52.189934 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189301    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:56:52.189934 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189304    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:56:52.189934 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189306    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:56:52.189934 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189309    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:56:52.189934 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189311    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:56:52.189934 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189314    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:56:52.189934 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189317    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:56:52.189934 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189319    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:56:52.189934 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189322    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:56:52.189934 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189324    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:56:52.189934 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189327    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:56:52.189934 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189330    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:56:52.189934 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189333    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:56:52.189934 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189336    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:56:52.189934 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189338    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:56:52.189934 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189341    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:56:52.189934 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189343    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:56:52.189934 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189346    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:56:52.189934 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189348    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:56:52.189934 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189351    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:56:52.190420 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189353    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:56:52.190420 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189356    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:56:52.190420 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189358    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:56:52.190420 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189361    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:56:52.190420 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189363    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:56:52.190420 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189368    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:56:52.190420 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189371    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:56:52.190420 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189374    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:56:52.190420 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189377    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:56:52.190420 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189380    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:56:52.190420 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189382    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:56:52.190420 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189385    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:56:52.190420 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189388    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:56:52.190420 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189391    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:56:52.190420 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189394    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:56:52.190420 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189396    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:56:52.190420 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189399    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:56:52.190420 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189401    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:56:52.190420 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189404    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:56:52.190420 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189406    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:56:52.190915 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189409    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:56:52.190915 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189411    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:56:52.190915 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189414    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:56:52.190915 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189416    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:56:52.190915 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189421    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:56:52.190915 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189424    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:56:52.190915 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189426    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:56:52.190915 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189429    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:56:52.190915 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189432    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:56:52.190915 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189435    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:56:52.190915 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189437    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:56:52.190915 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189440    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:56:52.190915 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189443    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:56:52.190915 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189446    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:56:52.190915 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189448    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:56:52.190915 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189451    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:56:52.190915 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189454    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:56:52.190915 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189458    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:56:52.190915 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189468    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:56:52.191426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189472    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:56:52.191426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189475    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:56:52.191426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189477    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:56:52.191426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189480    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:56:52.191426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189483    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:56:52.191426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189515    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:56:52.191426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189524    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:56:52.191426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189528    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:56:52.191426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189531    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:56:52.191426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189534    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:56:52.191426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189537    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:56:52.191426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189540    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:56:52.191426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189543    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:56:52.191426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189546    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:56:52.191426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.189549    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:56:52.191426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190306    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:56:52.191426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190313    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:56:52.191426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190317    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:56:52.191426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190320    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:56:52.191426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190323    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:56:52.191923 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190326    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:56:52.191923 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190329    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:56:52.191923 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190332    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:56:52.191923 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190336    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:56:52.191923 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190340    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:56:52.191923 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190343    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:56:52.191923 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190346    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:56:52.191923 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190349    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:56:52.191923 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190352    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:56:52.191923 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190355    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:56:52.191923 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190358    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:56:52.191923 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190361    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:56:52.191923 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190370    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:56:52.191923 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190373    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:56:52.191923 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190376    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:56:52.191923 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190379    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:56:52.191923 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190382    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:56:52.191923 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190384    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:56:52.191923 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190387    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:56:52.192402 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190389    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:56:52.192402 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190392    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:56:52.192402 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190395    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:56:52.192402 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190397    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:56:52.192402 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190400    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:56:52.192402 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190402    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:56:52.192402 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190405    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:56:52.192402 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190408    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:56:52.192402 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190411    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:56:52.192402 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190414    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:56:52.192402 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190417    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:56:52.192402 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190420    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:56:52.192402 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190423    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:56:52.192402 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190426    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:56:52.192402 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190429    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:56:52.192402 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190431    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:56:52.192402 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190434    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:56:52.192402 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190436    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:56:52.192402 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190439    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:56:52.192402 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190442    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:56:52.192888 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190444    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:56:52.192888 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190447    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:56:52.192888 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190449    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:56:52.192888 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190452    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:56:52.192888 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190455    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:56:52.192888 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190457    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:56:52.192888 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190465    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:56:52.192888 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190468    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:56:52.192888 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190471    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:56:52.192888 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190473    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:56:52.192888 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190476    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:56:52.192888 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190478    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:56:52.192888 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190481    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:56:52.192888 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190484    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:56:52.192888 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190486    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:56:52.192888 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190489    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:56:52.192888 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190491    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:56:52.192888 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190494    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:56:52.192888 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190499    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:56:52.193375 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190503    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:56:52.193375 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190506    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:56:52.193375 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190508    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:56:52.193375 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190511    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:56:52.193375 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190530    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:56:52.193375 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190534    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:56:52.193375 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190538    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:56:52.193375 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190541    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:56:52.193375 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190543    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:56:52.193375 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190546    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:56:52.193375 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190549    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:56:52.193375 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190552    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:56:52.193375 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190555    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:56:52.193375 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190557    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:56:52.193375 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190560    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:56:52.193375 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190563    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:56:52.193375 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190565    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:56:52.193375 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190568    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:56:52.193375 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190570    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:56:52.193375 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190573    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:56:52.193857 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190581    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:56:52.193857 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190584    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:56:52.193857 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190587    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:56:52.193857 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190595    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:56:52.193857 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190598    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:56:52.193857 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190600    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:56:52.193857 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190603    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:56:52.193857 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190608    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:56:52.193857 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190610    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:56:52.193857 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190614    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:56:52.193857 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190618    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:56:52.193857 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190621    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:56:52.193857 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190624    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:56:52.193857 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190626    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:56:52.193857 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190629    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:56:52.193857 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.190632    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:56:52.193857 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191036    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:56:52.193857 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191041    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:56:52.193857 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191043    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:56:52.193857 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191046    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:56:52.194365 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191049    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:56:52.194365 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191051    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:56:52.194365 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191054    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:56:52.194365 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191057    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:56:52.194365 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191059    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:56:52.194365 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191062    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:56:52.194365 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191064    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:56:52.194365 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191067    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:56:52.194365 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191069    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:56:52.194365 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191072    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:56:52.194365 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191074    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:56:52.194365 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191077    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:56:52.194365 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191083    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:56:52.194365 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191086    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:56:52.194365 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191088    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:56:52.194365 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191091    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:56:52.194365 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191093    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:56:52.194365 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191096    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:56:52.194365 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191099    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:56:52.194827 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191103    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:56:52.194827 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191105    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:56:52.194827 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191108    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:56:52.194827 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191113    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:56:52.194827 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191116    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:56:52.194827 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191119    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:56:52.194827 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191122    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:56:52.194827 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191127    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:56:52.194827 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191130    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:56:52.194827 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191133    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:56:52.194827 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191136    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:56:52.194827 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191138    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:56:52.194827 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191141    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:56:52.194827 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191143    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:56:52.194827 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191146    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:56:52.194827 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191148    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:56:52.194827 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191151    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:56:52.194827 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191161    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:56:52.194827 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191164    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:56:52.195312 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191167    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:56:52.195312 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191169    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:56:52.195312 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191173    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:56:52.195312 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191177    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:56:52.195312 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191182    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:56:52.195312 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191189    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:56:52.195312 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191192    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:56:52.195312 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191195    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:56:52.195312 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191198    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:56:52.195312 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191201    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:56:52.195312 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191203    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:56:52.195312 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191206    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:56:52.195312 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191209    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:56:52.195312 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191213    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:56:52.195312 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191215    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:56:52.195312 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191218    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:56:52.195312 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191221    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:56:52.195312 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191223    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:56:52.195312 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191226    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:56:52.195780 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191228    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:56:52.195780 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191232    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:56:52.195780 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191235    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:56:52.195780 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191237    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:56:52.195780 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191240    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:56:52.195780 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191243    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:56:52.195780 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191245    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:56:52.195780 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191248    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:56:52.195780 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191250    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:56:52.195780 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191253    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:56:52.195780 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191255    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:56:52.195780 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191258    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:56:52.195780 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191261    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:56:52.195780 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191263    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:56:52.195780 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191266    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:56:52.195780 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191268    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:56:52.195780 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191271    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:56:52.195780 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191273    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:56:52.195780 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191276    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:56:52.195780 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191278    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:56:52.196323 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191282    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:56:52.196323 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191285    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:56:52.196323 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191288    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:56:52.196323 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191291    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:56:52.196323 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191293    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:56:52.196323 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191296    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:56:52.196323 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191301    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:56:52.196323 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191304    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:56:52.196323 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191306    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:56:52.196323 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191308    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:56:52.196323 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191311    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:56:52.196323 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191313    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:56:52.196323 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191316    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:56:52.196323 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191320    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:56:52.196323 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191322    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:56:52.196323 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191325    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:56:52.196323 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191327    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:56:52.196323 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.191330    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:56:52.196323 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191401    2581 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 17:56:52.196323 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191447    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 17:56:52.196323 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191455    2581 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 17:56:52.196833 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191460    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 17:56:52.196833 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191464    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 17:56:52.196833 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191467    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 17:56:52.196833 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191471    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 17:56:52.196833 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191475    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 17:56:52.196833 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191478    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 17:56:52.196833 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191482    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 17:56:52.196833 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191486    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 17:56:52.196833 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191489    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 17:56:52.196833 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191492    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 17:56:52.196833 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191495    2581 flags.go:64] FLAG: --cgroup-root=""
Sep 17 17:56:52.196833 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191498    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 17:56:52.196833 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191501    2581 flags.go:64] FLAG: --client-ca-file=""
Sep 17 17:56:52.196833 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191505    2581 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 17:56:52.196833 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191509    2581 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 17:56:52.196833 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191513    2581 flags.go:64] FLAG: --cluster-domain=""
Sep 17 17:56:52.196833 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191515    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 17:56:52.196833 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191520    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 17:56:52.196833 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191524    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 17:56:52.196833 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191527    2581 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 17:56:52.196833 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191531    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 17:56:52.196833 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191534    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 17:56:52.196833 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191537    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 17:56:52.197394 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191541    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 17:56:52.197394 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191544    2581 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 17:56:52.197394 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191547    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 17:56:52.197394 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191550    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 17:56:52.197394 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191553    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 17:56:52.197394 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191557    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 17:56:52.197394 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191561    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 17:56:52.197394 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191564    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 17:56:52.197394 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191567    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 17:56:52.197394 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191570    2581 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 17:56:52.197394 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191573    2581 flags.go:64] FLAG: --enable-server="true"
Sep 17 17:56:52.197394 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191576    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 17:56:52.197394 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191580    2581 flags.go:64] FLAG: --event-burst="100"
Sep 17 17:56:52.197394 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191583    2581 flags.go:64] FLAG: --event-qps="50"
Sep 17 17:56:52.197394 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191586    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 17:56:52.197394 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191589    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 17:56:52.197394 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191592    2581 flags.go:64] FLAG: --eviction-hard=""
Sep 17 17:56:52.197394 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191596    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 17:56:52.197394 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191599    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 17:56:52.197394 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191602    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 17:56:52.197394 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191605    2581 flags.go:64] FLAG: --eviction-soft=""
Sep 17 17:56:52.197394 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191608    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 17:56:52.197394 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191610    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 17:56:52.197394 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191614    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 17:56:52.197394 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191617    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 17:56:52.198031 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191619    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 17:56:52.198031 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191624    2581 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 17:56:52.198031 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191627    2581 flags.go:64] FLAG: --feature-gates=""
Sep 17 17:56:52.198031 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191631    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 17:56:52.198031 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191634    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 17:56:52.198031 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191637    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 17:56:52.198031 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191640    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 17:56:52.198031 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191643    2581 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 17:56:52.198031 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191647    2581 flags.go:64] FLAG: --help="false"
Sep 17 17:56:52.198031 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191649    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-130-28.ec2.internal"
Sep 17 17:56:52.198031 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191653    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 17:56:52.198031 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191656    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 17:56:52.198031 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191659    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 17:56:52.198031 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191662    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 17:56:52.198031 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191666    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 17:56:52.198031 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191669    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 17:56:52.198031 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191672    2581 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 17:56:52.198031 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191674    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 17:56:52.198031 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191677    2581 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 17:56:52.198031 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191680    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 17:56:52.198031 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191683    2581 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 17:56:52.198031 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191686    2581 flags.go:64] FLAG: --kube-reserved=""
Sep 17 17:56:52.198031 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191689    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 17:56:52.198031 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191692    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 17:56:52.198604 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191695    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 17:56:52.198604 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191697    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 17:56:52.198604 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191700    2581 flags.go:64] FLAG: --lock-file=""
Sep 17 17:56:52.198604 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191703    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 17:56:52.198604 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191706    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 17:56:52.198604 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191709    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 17:56:52.198604 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191714    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 17:56:52.198604 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191718    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 17:56:52.198604 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191721    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 17:56:52.198604 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191725    2581 flags.go:64] FLAG: --logging-format="text"
Sep 17 17:56:52.198604 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191727    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 17:56:52.198604 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191731    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 17:56:52.198604 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191733    2581 flags.go:64] FLAG: --manifest-url=""
Sep 17 17:56:52.198604 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191737    2581 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 17:56:52.198604 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191741    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 17:56:52.198604 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191744    2581 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 17:56:52.198604 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191749    2581 flags.go:64] FLAG: --max-pods="110"
Sep 17 17:56:52.198604 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191752    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 17:56:52.198604 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191755    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 17:56:52.198604 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191758    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 17:56:52.198604 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191761    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 17:56:52.198604 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191764    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 17:56:52.198604 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191767    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 17:56:52.198604 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191770    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 17:56:52.198604 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191777    2581 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 17:56:52.199238 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191780    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 17:56:52.199238 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191783    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 17:56:52.199238 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191786    2581 flags.go:64] FLAG: --pod-cidr=""
Sep 17 17:56:52.199238 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191789    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 17:56:52.199238 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191794    2581 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 17:56:52.199238 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191797    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 17:56:52.199238 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191800    2581 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 17:56:52.199238 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191803    2581 flags.go:64] FLAG: --port="10250"
Sep 17 17:56:52.199238 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191806    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 17:56:52.199238 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191809    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0adaa47c1c782083e"
Sep 17 17:56:52.199238 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191812    2581 flags.go:64] FLAG: --qos-reserved=""
Sep 17 17:56:52.199238 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191815    2581 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 17:56:52.199238 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191818    2581 flags.go:64] FLAG: --register-node="true"
Sep 17 17:56:52.199238 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191821    2581 flags.go:64] FLAG: --register-with-taints=""
Sep 17 17:56:52.199238 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191825    2581 flags.go:64] FLAG: --registry-burst="10"
Sep 17 17:56:52.199238 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191828    2581 flags.go:64] FLAG: --registry-qps="5"
Sep 17 17:56:52.199238 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191831    2581 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 17:56:52.199238 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191835    2581 flags.go:64] FLAG: --reserved-memory=""
Sep 17 17:56:52.199238 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191839    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 17:56:52.199238 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191842    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 17:56:52.199238 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191845    2581 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 17:56:52.199238 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191848    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 17:56:52.199238 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191851    2581 flags.go:64] FLAG: --runonce="false"
Sep 17 17:56:52.199238 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191854    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 17:56:52.199238 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191857    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 17:56:52.199868 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191860    2581 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 17:56:52.199868 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191863    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 17:56:52.199868 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191866    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 17:56:52.199868 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191869    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 17:56:52.199868 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191872    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 17:56:52.199868 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191875    2581 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 17:56:52.199868 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191878    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 17:56:52.199868 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191881    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 17:56:52.199868 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191884    2581 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 17:56:52.199868 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191887    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 17:56:52.199868 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191890    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 17:56:52.199868 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191893    2581 flags.go:64] FLAG: --system-cgroups=""
Sep 17 17:56:52.199868 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191895    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 17:56:52.199868 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191912    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 17:56:52.199868 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191915    2581 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 17:56:52.199868 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191918    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 17:56:52.199868 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191922    2581 flags.go:64] FLAG: --tls-min-version=""
Sep 17 17:56:52.199868 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191925    2581 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 17:56:52.199868 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191928    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 17:56:52.199868 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191931    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 17:56:52.199868 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191935    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 17:56:52.199868 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191938    2581 flags.go:64] FLAG: --v="2"
Sep 17 17:56:52.199868 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191942    2581 flags.go:64] FLAG: --version="false"
Sep 17 17:56:52.199868 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191947    2581 flags.go:64] FLAG: --vmodule=""
Sep 17 17:56:52.199868 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191952    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 17:56:52.200514 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.191956    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 17:56:52.200514 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192068    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:56:52.200514 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192072    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:56:52.200514 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192074    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:56:52.200514 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192079    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:56:52.200514 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192082    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:56:52.200514 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192086    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:56:52.200514 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192088    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:56:52.200514 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192091    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:56:52.200514 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192094    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:56:52.200514 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192097    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:56:52.200514 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192099    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:56:52.200514 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192102    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:56:52.200514 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192104    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:56:52.200514 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192107    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:56:52.200514 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192110    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:56:52.200514 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192112    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:56:52.200514 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192115    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:56:52.200514 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192118    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:56:52.200514 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192120    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:56:52.200514 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192123    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:56:52.201035 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192126    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:56:52.201035 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192128    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:56:52.201035 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192131    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:56:52.201035 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192134    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:56:52.201035 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192136    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:56:52.201035 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192139    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:56:52.201035 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192142    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:56:52.201035 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192145    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:56:52.201035 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192147    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:56:52.201035 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192150    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:56:52.201035 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192154    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:56:52.201035 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192157    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:56:52.201035 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192159    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:56:52.201035 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192162    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:56:52.201035 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192164    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:56:52.201035 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192168    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:56:52.201035 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192171    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:56:52.201035 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192173    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:56:52.201035 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192176    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:56:52.201035 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192179    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:56:52.201524 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192181    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:56:52.201524 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192184    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:56:52.201524 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192187    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:56:52.201524 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192189    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:56:52.201524 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192192    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:56:52.201524 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192195    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:56:52.201524 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192197    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:56:52.201524 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192200    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:56:52.201524 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192203    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:56:52.201524 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192205    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:56:52.201524 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192208    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:56:52.201524 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192211    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:56:52.201524 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192213    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:56:52.201524 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192216    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:56:52.201524 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192218    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:56:52.201524 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192221    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:56:52.201524 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192224    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:56:52.201524 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192226    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:56:52.201524 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192229    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:56:52.201524 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192232    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:56:52.202027 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192234    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:56:52.202027 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192236    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:56:52.202027 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192240    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:56:52.202027 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192244    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:56:52.202027 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192246    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:56:52.202027 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192249    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:56:52.202027 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192251    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:56:52.202027 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192255    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:56:52.202027 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192257    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:56:52.202027 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192260    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:56:52.202027 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192262    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:56:52.202027 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192265    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:56:52.202027 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192268    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:56:52.202027 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192270    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:56:52.202027 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192273    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:56:52.202027 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192277    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:56:52.202027 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192281    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:56:52.202027 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192284    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:56:52.202027 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192287    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:56:52.202503 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192290    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:56:52.202503 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192293    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:56:52.202503 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192296    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:56:52.202503 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192300    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:56:52.202503 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192304    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:56:52.202503 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192307    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:56:52.202503 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192310    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:56:52.202503 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192312    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:56:52.202503 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192316    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:56:52.202503 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192319    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:56:52.202503 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192322    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:56:52.202503 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192325    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:56:52.202503 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192328    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:56:52.202503 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192331    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:56:52.202503 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192334    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:56:52.202503 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192338    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:56:52.202503 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192341    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:56:52.202503 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192344    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:56:52.202503 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192347    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:56:52.203067 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.192350    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:56:52.203067 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.192868    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:56:52.203067 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.199382    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 17:56:52.203067 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.199398    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 17:56:52.203067 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.199416    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 17:56:52.203067 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.199422    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 17:56:52.203067 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199475    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:56:52.203067 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199480    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:56:52.203067 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199484    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:56:52.203067 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199487    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:56:52.203067 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199490    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:56:52.203067 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199492    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:56:52.203067 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199495    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:56:52.203067 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199498    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:56:52.203426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199502    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:56:52.203426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199505    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:56:52.203426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199508    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:56:52.203426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199511    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:56:52.203426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199514    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:56:52.203426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199517    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:56:52.203426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199520    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:56:52.203426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199523    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:56:52.203426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199525    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:56:52.203426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199528    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:56:52.203426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199531    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:56:52.203426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199534    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:56:52.203426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199537    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:56:52.203426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199540    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:56:52.203426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199543    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:56:52.203426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199546    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:56:52.203426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199549    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:56:52.203426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199552    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:56:52.203426 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199555    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:56:52.203876 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199557    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:56:52.203876 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199560    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:56:52.203876 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199564    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:56:52.203876 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199567    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:56:52.203876 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199569    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:56:52.203876 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199572    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:56:52.203876 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199574    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:56:52.203876 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199577    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:56:52.203876 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199580    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:56:52.203876 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199583    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:56:52.203876 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199585    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:56:52.203876 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199588    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:56:52.203876 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199590    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:56:52.203876 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199593    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:56:52.203876 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199596    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:56:52.203876 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199598    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:56:52.203876 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199601    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:56:52.203876 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199603    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:56:52.203876 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199606    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:56:52.203876 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199609    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:56:52.204377 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199611    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:56:52.204377 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199614    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:56:52.204377 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199616    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:56:52.204377 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199619    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:56:52.204377 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199622    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:56:52.204377 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199624    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:56:52.204377 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199627    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:56:52.204377 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199629    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:56:52.204377 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199633    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:56:52.204377 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199637    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:56:52.204377 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199639    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:56:52.204377 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199642    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:56:52.204377 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199646    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:56:52.204377 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199650    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:56:52.204377 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199654    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:56:52.204377 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199657    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:56:52.204377 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199660    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:56:52.204377 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199662    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:56:52.204377 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199665    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:56:52.204840 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199667    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:56:52.204840 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199670    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:56:52.204840 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199672    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:56:52.204840 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199675    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:56:52.204840 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199678    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:56:52.204840 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199680    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:56:52.204840 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199683    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:56:52.204840 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199685    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:56:52.204840 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199688    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:56:52.204840 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199691    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:56:52.204840 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199693    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:56:52.204840 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199696    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:56:52.204840 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199698    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:56:52.204840 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199701    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:56:52.204840 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199704    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:56:52.204840 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199706    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:56:52.204840 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199709    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:56:52.204840 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199712    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:56:52.204840 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199716    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:56:52.204840 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199719    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:56:52.205338 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199722    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:56:52.205338 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199725    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:56:52.205338 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199728    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:56:52.205338 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199731    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:56:52.205338 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199734    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:56:52.205338 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199737    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:56:52.205338 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199739    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:56:52.205338 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199742    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:56:52.205338 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199745    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:56:52.205338 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199748    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:56:52.205338 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199751    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:56:52.205338 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199753    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:56:52.205338 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199756    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:56:52.205338 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.199760    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:56:52.205338 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199870    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:56:52.205705 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199874    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:56:52.205705 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199878    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:56:52.205705 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199881    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:56:52.205705 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199884    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:56:52.205705 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199887    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:56:52.205705 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199890    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:56:52.205705 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199892    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:56:52.205705 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199895    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:56:52.205705 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199913    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:56:52.205705 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199917    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:56:52.205705 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199920    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:56:52.205705 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199922    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:56:52.205705 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199925    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:56:52.205705 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199928    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:56:52.205705 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199931    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:56:52.205705 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199935    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:56:52.205705 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199939    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:56:52.205705 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199942    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:56:52.205705 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199945    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:56:52.205705 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199948    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:56:52.206244 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199951    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:56:52.206244 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199953    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:56:52.206244 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199956    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:56:52.206244 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199959    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:56:52.206244 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199962    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:56:52.206244 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199964    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:56:52.206244 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199967    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:56:52.206244 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199971    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:56:52.206244 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199973    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:56:52.206244 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199976    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:56:52.206244 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199979    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:56:52.206244 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199981    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:56:52.206244 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199984    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:56:52.206244 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199986    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:56:52.206244 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199989    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:56:52.206244 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199991    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:56:52.206244 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199994    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:56:52.206244 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.199997    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:56:52.206244 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200000    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:56:52.206244 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200004    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:56:52.206723 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200009    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:56:52.206723 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200014    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:56:52.206723 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200018    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:56:52.206723 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200021    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:56:52.206723 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200024    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:56:52.206723 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200026    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:56:52.206723 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200029    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:56:52.206723 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200032    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:56:52.206723 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200035    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:56:52.206723 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200037    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:56:52.206723 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200040    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:56:52.206723 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200043    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:56:52.206723 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200045    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:56:52.206723 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200049    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:56:52.206723 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200052    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:56:52.206723 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200055    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:56:52.206723 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200058    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:56:52.206723 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200060    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:56:52.206723 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200063    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:56:52.206723 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200065    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:56:52.207313 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200069    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:56:52.207313 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200072    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:56:52.207313 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200074    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:56:52.207313 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200077    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:56:52.207313 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200079    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:56:52.207313 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200082    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:56:52.207313 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200085    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:56:52.207313 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200087    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:56:52.207313 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200090    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:56:52.207313 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200092    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:56:52.207313 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200094    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:56:52.207313 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200097    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:56:52.207313 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200100    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:56:52.207313 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200103    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:56:52.207313 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200105    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:56:52.207313 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200108    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:56:52.207313 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200111    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:56:52.207313 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200113    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:56:52.207313 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200116    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:56:52.207767 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200118    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:56:52.207767 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200121    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:56:52.207767 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200123    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:56:52.207767 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200126    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:56:52.207767 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200128    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:56:52.207767 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200131    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:56:52.207767 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200134    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:56:52.207767 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200136    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:56:52.207767 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200139    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:56:52.207767 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200142    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:56:52.207767 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200144    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:56:52.207767 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200146    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:56:52.207767 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200149    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:56:52.207767 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200151    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:56:52.207767 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200154    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:56:52.207767 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200157    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:56:52.207767 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200159    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:56:52.207767 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200162    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:56:52.207767 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:52.200165    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:56:52.208234 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.200171    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:56:52.208234 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.200271    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 17:56:52.208234 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.202091    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 17:56:52.208234 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.203028    2581 server.go:1019] "Starting client certificate rotation"
Sep 17 17:56:52.208234 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.203126    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:56:52.208234 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.203166    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:56:52.208234 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.205980    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:56:52.213559 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.213538    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:56:52.224589 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.224570    2581 log.go:25] "Validated CRI v1 runtime API"
Sep 17 17:56:52.226149 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.226135    2581 log.go:25] "Validated CRI v1 image API"
Sep 17 17:56:52.227011 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.226997    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 17:56:52.232091 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.232070    2581 fs.go:135] Filesystem UUIDs: map[2a979d22-4008-4c0e-9d8d-f92a1417281d:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 832a1a90-33fd-4722-8df5-0b4081400833:/dev/nvme0n1p3]
Sep 17 17:56:52.232163 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.232089    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 17:56:52.235552 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.235509    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:56:52.237361 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.237251    2581 manager.go:216] Machine: {Timestamp:2026-09-17 17:56:52.236006486 +0000 UTC m=+0.326628128 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3132189 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2db9f27a3e82963fc08f32cf5965ed SystemUUID:ec2db9f2-7a3e-8296-3fc0-8f32cf5965ed BootID:ef0cb8d6-9812-414a-9a9c-b3888d4581e9 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f7:8f:8c:95 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f7:8f:8c:95 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f2:c3:b6:10:fb:58 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 17:56:52.237361 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.237356    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 17:56:52.237489 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.237441    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 17:56:52.241357 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.241330    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 17:56:52.241495 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.241360    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-28.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 17:56:52.242070 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.242059    2581 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 17:56:52.242107 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.242073    2581 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 17:56:52.242107 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.242081    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:56:52.242709 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.242699    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:56:52.242779 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.242771    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 17:56:52.245304 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.245293    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:56:52.245416 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.245406    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 17:56:52.245454 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.245436    2581 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 17:56:52.245454 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.245451    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 17:56:52.245509 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.245463    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 17:56:52.245509 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.245473    2581 kubelet.go:398] "Adding apiserver pod source"
Sep 17 17:56:52.245509 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.245481    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 17:56:52.245829 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.245817    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:56:52.245878 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.245835    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:56:52.247914 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.247888    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 17:56:52.249842 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.249829    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 17:56:52.249884 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.249860    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 17:56:52.250873 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.250860    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 17:56:52.250946 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.250877    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 17:56:52.250946 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.250883    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 17:56:52.250946 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.250889    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 17:56:52.250946 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.250914    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 17:56:52.250946 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.250922    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 17:56:52.250946 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.250945    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 17:56:52.251126 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.250951    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 17:56:52.251126 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.250958    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 17:56:52.251126 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.250963    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 17:56:52.251126 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.250980    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 17:56:52.251126 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.250985    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 17:56:52.251576 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.251566    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 17:56:52.251618 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.251576    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 17:56:52.255287 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.255272    2581 server.go:1268] "Started kubelet"
Sep 17 17:56:52.255376 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.255344    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 17:56:52.258849 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.258673    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 17:56:52.258937 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.258881    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 17:56:52.259180 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:52.259145    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 17:56:52.259313 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:52.259239    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-28.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 17:56:52.259402 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.259374    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-28.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 17:56:52.259804 ip-10-0-130-28 systemd[1]: Started Kubernetes Kubelet.
Sep 17 17:56:52.262221 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.262207    2581 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 17:56:52.265581 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.265566    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 17:56:52.266950 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:52.266086    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-28.ec2.internal.18d62d64a76c9c24  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-28.ec2.internal,UID:ip-10-0-130-28.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-28.ec2.internal,},FirstTimestamp:2026-09-17 17:56:52.2552433 +0000 UTC m=+0.345864942,LastTimestamp:2026-09-17 17:56:52.2552433 +0000 UTC m=+0.345864942,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-28.ec2.internal,}"
Sep 17 17:56:52.267185 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:52.267169    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 17:56:52.270406 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.270387    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 17:56:52.270962 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.270946    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 17:56:52.271065 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:52.271020    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found"
Sep 17 17:56:52.271409 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.271396    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 17:56:52.271485 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.271396    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 17:56:52.271485 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.271426    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 17:56:52.271608 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.271506    2581 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 17:56:52.271608 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.271517    2581 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 17:56:52.271727 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.271713    2581 factory.go:55] Registering systemd factory
Sep 17 17:56:52.271771 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.271728    2581 factory.go:223] Registration of the systemd container factory successfully
Sep 17 17:56:52.271955 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.271941    2581 factory.go:153] Registering CRI-O factory
Sep 17 17:56:52.271955 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.271956    2581 factory.go:223] Registration of the crio container factory successfully
Sep 17 17:56:52.272067 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.271993    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 17:56:52.272067 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.272007    2581 factory.go:103] Registering Raw factory
Sep 17 17:56:52.272067 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.272018    2581 manager.go:1196] Started watching for new ooms in manager
Sep 17 17:56:52.272278 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.272267    2581 manager.go:349] Starting recovery of all containers
Sep 17 17:56:52.274912 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:52.274877    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 17:56:52.275018 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:52.274896    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-28.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 17:56:52.283427 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.283298    2581 manager.go:354] Recovery completed
Sep 17 17:56:52.287419 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.287404    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:56:52.290741 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.290718    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:56:52.290805 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.290754    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:56:52.290805 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.290767    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:56:52.291357 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.291344    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:56:52.291357 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.291347    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 17:56:52.291504 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.291362    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 17:56:52.291504 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.291377    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:56:52.292182 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.292165    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:56:52.292256 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.292193    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:56:52.292256 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.292205    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:56:52.292810 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:52.292730    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-28.ec2.internal.18d62d64a98a42ee  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-28.ec2.internal,UID:ip-10-0-130-28.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-28.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-28.ec2.internal,},FirstTimestamp:2026-09-17 17:56:52.290740974 +0000 UTC m=+0.381362614,LastTimestamp:2026-09-17 17:56:52.290740974 +0000 UTC m=+0.381362614,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-28.ec2.internal,}"
Sep 17 17:56:52.292913 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.292798    2581 policy_none.go:49] "None policy: Start"
Sep 17 17:56:52.292913 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.292892    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 17:56:52.293024 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.292917    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 17:56:52.294233 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.294218    2581 policy_none.go:47] "Start"
Sep 17 17:56:52.304749 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:52.304688    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-28.ec2.internal.18d62d64a98a8a3b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-28.ec2.internal,UID:ip-10-0-130-28.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-28.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-28.ec2.internal,},FirstTimestamp:2026-09-17 17:56:52.290759227 +0000 UTC m=+0.381380869,LastTimestamp:2026-09-17 17:56:52.290759227 +0000 UTC m=+0.381380869,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-28.ec2.internal,}"
Sep 17 17:56:52.314265 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:52.314201    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-28.ec2.internal.18d62d64a98ab87e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-28.ec2.internal,UID:ip-10-0-130-28.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-28.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-28.ec2.internal,},FirstTimestamp:2026-09-17 17:56:52.29077107 +0000 UTC m=+0.381392712,LastTimestamp:2026-09-17 17:56:52.29077107 +0000 UTC m=+0.381392712,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-28.ec2.internal,}"
Sep 17 17:56:52.327522 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:52.327465    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-28.ec2.internal.18d62d64a98a42ee\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-28.ec2.internal.18d62d64a98a42ee  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-28.ec2.internal,UID:ip-10-0-130-28.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-28.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-28.ec2.internal,},FirstTimestamp:2026-09-17 17:56:52.290740974 +0000 UTC m=+0.381362614,LastTimestamp:2026-09-17 17:56:52.292180232 +0000 UTC m=+0.382801873,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-28.ec2.internal,}"
Sep 17 17:56:52.328280 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.328260    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2d2zf"
Sep 17 17:56:52.328477 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.328465    2581 manager.go:337] "Starting Device Plugin manager"
Sep 17 17:56:52.328520 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:52.328509    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 17:56:52.328569 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.328521    2581 server.go:87] "Starting device plugin registration server"
Sep 17 17:56:52.328837 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.328825    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 17:56:52.328893 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.328837    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 17:56:52.328990 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.328975    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 17:56:52.329149 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.329068    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 17:56:52.329149 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.329080    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 17:56:52.329544 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:52.329514    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 17:56:52.329688 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:52.329561    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-28.ec2.internal\" not found"
Sep 17 17:56:52.338498 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.338482    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2d2zf"
Sep 17 17:56:52.350448 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.350428    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 17:56:52.351539 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.351525    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 17:56:52.351600 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.351575    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 17:56:52.351600 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.351598    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 17:56:52.351687 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:52.351639    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 17:56:52.354752 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.354734    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:56:52.429638 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.429601    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:56:52.430706 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.430688    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:56:52.430800 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.430723    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:56:52.430800 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.430735    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:56:52.430800 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.430763    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-28.ec2.internal"
Sep 17 17:56:52.448960 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.448875    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-28.ec2.internal"
Sep 17 17:56:52.448960 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:52.448897    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-28.ec2.internal\": node \"ip-10-0-130-28.ec2.internal\" not found"
Sep 17 17:56:52.451741 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.451723    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal"]
Sep 17 17:56:52.451802 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.451786    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:56:52.452603 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.452589    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:56:52.452665 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.452615    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:56:52.452665 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.452624    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:56:52.456277 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.456265    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:56:52.456415 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.456396    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal"
Sep 17 17:56:52.456474 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.456438    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:56:52.456985 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.456970    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:56:52.457067 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.456995    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:56:52.457067 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.457010    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:56:52.457067 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.457034    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:56:52.457067 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.457057    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:56:52.457244 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.457071    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:56:52.458895 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.458873    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal"
Sep 17 17:56:52.458995 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.458917    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:56:52.459607 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.459591    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:56:52.459671 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.459620    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:56:52.459671 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.459635    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:56:52.472585 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.472568    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d845ebff585bafdc97c5cabd0509365e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal\" (UID: \"d845ebff585bafdc97c5cabd0509365e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal"
Sep 17 17:56:52.472666 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.472598    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d845ebff585bafdc97c5cabd0509365e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal\" (UID: \"d845ebff585bafdc97c5cabd0509365e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal"
Sep 17 17:56:52.472666 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.472623    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/672f4fffca246081b3733694dc0095b6-config\") pod \"kube-apiserver-proxy-ip-10-0-130-28.ec2.internal\" (UID: \"672f4fffca246081b3733694dc0095b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal"
Sep 17 17:56:52.474513 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:52.474495    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-28.ec2.internal\" not found" node="ip-10-0-130-28.ec2.internal"
Sep 17 17:56:52.477745 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:52.477731    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-28.ec2.internal\" not found" node="ip-10-0-130-28.ec2.internal"
Sep 17 17:56:52.494457 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:52.494440    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found"
Sep 17 17:56:52.573065 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.573045    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/672f4fffca246081b3733694dc0095b6-config\") pod \"kube-apiserver-proxy-ip-10-0-130-28.ec2.internal\" (UID: \"672f4fffca246081b3733694dc0095b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal"
Sep 17 17:56:52.573157 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.573070    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d845ebff585bafdc97c5cabd0509365e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal\" (UID: \"d845ebff585bafdc97c5cabd0509365e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal"
Sep 17 17:56:52.573157 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.573085    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d845ebff585bafdc97c5cabd0509365e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal\" (UID: \"d845ebff585bafdc97c5cabd0509365e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal"
Sep 17 17:56:52.573157 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.573126    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d845ebff585bafdc97c5cabd0509365e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal\" (UID: \"d845ebff585bafdc97c5cabd0509365e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal"
Sep 17 17:56:52.573274 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.573153    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d845ebff585bafdc97c5cabd0509365e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal\" (UID: \"d845ebff585bafdc97c5cabd0509365e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal"
Sep 17 17:56:52.573274 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.573157    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/672f4fffca246081b3733694dc0095b6-config\") pod \"kube-apiserver-proxy-ip-10-0-130-28.ec2.internal\" (UID: \"672f4fffca246081b3733694dc0095b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal"
Sep 17 17:56:52.594520 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:52.594502    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found"
Sep 17 17:56:52.694586 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:52.694558    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found"
Sep 17 17:56:52.777886 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.777825    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal"
Sep 17 17:56:52.780785 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:52.780766    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal"
Sep 17 17:56:52.795615 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:52.795593    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found"
Sep 17 17:56:52.896532 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:52.896500    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found"
Sep 17 17:56:52.996996 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:52.996961    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found"
Sep 17 17:56:53.097491 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:53.097421    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found"
Sep 17 17:56:53.197945 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:53.197914    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found"
Sep 17 17:56:53.203028 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:53.203007    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 17:56:53.203174 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:53.203148    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:56:53.275782 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:53.275749    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:56:53.298044 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:53.298018    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found"
Sep 17 17:56:53.342510 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:53.342462    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 17:51:52 +0000 UTC" deadline="2028-04-01 01:28:21.705721843 +0000 UTC"
Sep 17 17:56:53.342510 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:53.342500    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13471h31m28.363226844s"
Sep 17 17:56:53.368028 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:53.367955    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:56:53.368996 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:53.368976    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:56:53.369062 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:53.369011    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:56:53.369062 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:53.369024    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:56:53.399139 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:53.399118    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found"
Sep 17 17:56:53.427361 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:53.427326    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod672f4fffca246081b3733694dc0095b6.slice/crio-3cb27b787512b26fdd4a920ed133697e2d0a1cdce51589dd00fc1ad3499a6a67 WatchSource:0}: Error finding container 3cb27b787512b26fdd4a920ed133697e2d0a1cdce51589dd00fc1ad3499a6a67: Status 404 returned error can't find the container with id 3cb27b787512b26fdd4a920ed133697e2d0a1cdce51589dd00fc1ad3499a6a67
Sep 17 17:56:53.428054 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:53.428029    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd845ebff585bafdc97c5cabd0509365e.slice/crio-71ffd37ef4690c8b9f93403bf0af1a96c767767cad15ce7669f8bb748670b267 WatchSource:0}: Error finding container 71ffd37ef4690c8b9f93403bf0af1a96c767767cad15ce7669f8bb748670b267: Status 404 returned error can't find the container with id 71ffd37ef4690c8b9f93403bf0af1a96c767767cad15ce7669f8bb748670b267
Sep 17 17:56:53.435635 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:53.435621    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:56:53.500257 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:53.500226    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found"
Sep 17 17:56:53.569879 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:53.569849    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:56:53.601247 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:53.601218    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-28.ec2.internal\" not found"
Sep 17 17:56:53.686058 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:53.685979    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:56:53.771636 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:53.771602    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal"
Sep 17 17:56:53.790502 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:53.790474    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:56:53.792640 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:53.792616    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal"
Sep 17 17:56:53.811379 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:53.811357    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:56:54.163648 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.163553    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:56:54.247429 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.247397    2581 apiserver.go:52] "Watching apiserver"
Sep 17 17:56:54.256259 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.256230    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 17:56:54.257603 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.257566    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-24pww","openshift-network-diagnostics/network-check-target-z7848","openshift-ovn-kubernetes/ovnkube-node-bz9ng","kube-system/global-pull-secret-syncer-9xft5","kube-system/konnectivity-agent-8tpnw","kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal","openshift-dns/node-resolver-qcqn9","openshift-image-registry/node-ca-zsz9h","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal","openshift-network-operator/iptables-alerter-z2vvf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plwn4","openshift-cluster-node-tuning-operator/tuned-2c98r","openshift-multus/multus-4nrxj","openshift-multus/multus-additional-cni-plugins-v6f7b"]
Sep 17 17:56:54.260198 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.260175    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z2vvf"
Sep 17 17:56:54.263602 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.263575    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.266109 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.266086    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.268724 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.268698    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v6f7b"
Sep 17 17:56:54.270581 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.270545    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 17:56:54.270581 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.270574    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:56:54.271578 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.271177    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8tpnw"
Sep 17 17:56:54.271578 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.271420    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 17:56:54.271578 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.271177    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-srmt7\""
Sep 17 17:56:54.273703 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.273682    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plwn4"
Sep 17 17:56:54.275894 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.274990    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 17:56:54.275894 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.275199    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 17:56:54.275894 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.275357    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 17:56:54.275894 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.275397    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 17:56:54.275894 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.275425    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 17:56:54.275894 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.275577    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 17:56:54.275894 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.275603    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 17:56:54.275894 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.275654    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 17:56:54.275894 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.275698    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 17:56:54.275894 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.275814    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 17:56:54.276505 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.275889    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zch5c\""
Sep 17 17:56:54.276505 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.276368    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:56:54.276505 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.276424    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 17:56:54.276505 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:54.276461    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24pww" podUID="87bce58a-35d3-497c-8d82-7f925fe82b76"
Sep 17 17:56:54.276757 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.276628    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 17:56:54.276875 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.276835    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gvbr4\""
Sep 17 17:56:54.278254 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.277131    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 17:56:54.278254 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.277182    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2ktng\""
Sep 17 17:56:54.278254 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.277361    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 17:56:54.278254 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.277611    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lwgbm\""
Sep 17 17:56:54.278254 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.278060    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-f2xmg\""
Sep 17 17:56:54.278254 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.278091    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 17:56:54.278254 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.278131    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 17:56:54.279624 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.279600    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qcqn9"
Sep 17 17:56:54.279726 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.279701    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 17:56:54.279975 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.279957    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 17:56:54.283002 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.282580    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/041688ba-3c1a-426b-84b3-ec168a6a8658-ovnkube-config\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.283002 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.282627    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-cnibin\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.283002 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.282649    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-cni-binary-copy\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.283002 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.282675    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-host-slash\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.283002 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.282713    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-host-run-netns\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.283002 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.282746    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/83940065-1b61-4aca-819e-102e270dffc4-cnibin\") pod \"multus-additional-cni-plugins-v6f7b\" (UID: \"83940065-1b61-4aca-819e-102e270dffc4\") " pod="openshift-multus/multus-additional-cni-plugins-v6f7b"
Sep 17 17:56:54.283002 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.282789    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s68kf\" (UniqueName: \"kubernetes.io/projected/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-kube-api-access-s68kf\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.283002 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.282865    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-node-log\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.283002 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.282902    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/83940065-1b61-4aca-819e-102e270dffc4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v6f7b\" (UID: \"83940065-1b61-4aca-819e-102e270dffc4\") " pod="openshift-multus/multus-additional-cni-plugins-v6f7b"
Sep 17 17:56:54.283499 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.283068    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-host-var-lib-cni-bin\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.283499 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.283186    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.283499 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.283239    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0f02a062-ac1a-4caf-a488-3543ce36210f-etc-selinux\") pod \"aws-ebs-csi-driver-node-plwn4\" (UID: \"0f02a062-ac1a-4caf-a488-3543ce36210f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plwn4"
Sep 17 17:56:54.283499 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.283288    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-host-run-k8s-cni-cncf-io\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.283499 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.283339    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-multus-conf-dir\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.283499 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.283382    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-multus-cni-dir\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.283804 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.283506    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-etc-openvswitch\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.283804 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.283565    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-host-cni-netd\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.283804 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.283618    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/041688ba-3c1a-426b-84b3-ec168a6a8658-env-overrides\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.283804 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.283659    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0f02a062-ac1a-4caf-a488-3543ce36210f-socket-dir\") pod \"aws-ebs-csi-driver-node-plwn4\" (UID: \"0f02a062-ac1a-4caf-a488-3543ce36210f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plwn4"
Sep 17 17:56:54.283804 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.283699    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qd97\" (UniqueName: \"kubernetes.io/projected/0f02a062-ac1a-4caf-a488-3543ce36210f-kube-api-access-6qd97\") pod \"aws-ebs-csi-driver-node-plwn4\" (UID: \"0f02a062-ac1a-4caf-a488-3543ce36210f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plwn4"
Sep 17 17:56:54.283804 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.283773    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr78c\" (UniqueName: \"kubernetes.io/projected/c1b1a6b8-52b7-45fb-9f31-5533cc2c85e8-kube-api-access-sr78c\") pod \"iptables-alerter-z2vvf\" (UID: \"c1b1a6b8-52b7-45fb-9f31-5533cc2c85e8\") " pod="openshift-network-operator/iptables-alerter-z2vvf"
Sep 17 17:56:54.284110 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.283811    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-var-lib-openvswitch\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.284110 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.283861    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-run-ovn\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.284110 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.283869    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 17:56:54.284110 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.283894    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-host-cni-bin\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.284110 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.283963    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xch9q\" (UniqueName: \"kubernetes.io/projected/041688ba-3c1a-426b-84b3-ec168a6a8658-kube-api-access-xch9q\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.284342 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.284034    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0f02a062-ac1a-4caf-a488-3543ce36210f-registration-dir\") pod \"aws-ebs-csi-driver-node-plwn4\" (UID: \"0f02a062-ac1a-4caf-a488-3543ce36210f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plwn4"
Sep 17 17:56:54.284342 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.284288    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-host-var-lib-kubelet\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.284342 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.284333    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-host-kubelet\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.284487 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.284372    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0f73bdd3-5b33-4d88-9d8b-eb540633a58e-konnectivity-ca\") pod \"konnectivity-agent-8tpnw\" (UID: \"0f73bdd3-5b33-4d88-9d8b-eb540633a58e\") " pod="kube-system/konnectivity-agent-8tpnw"
Sep 17 17:56:54.284487 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.284441    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-h7lph\""
Sep 17 17:56:54.284600 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.284582    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 17:56:54.284658 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.284597    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-run-systemd\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.284712 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.284664    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/041688ba-3c1a-426b-84b3-ec168a6a8658-ovnkube-script-lib\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.284712 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.284704    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-system-cni-dir\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.285491 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.284769    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c1b1a6b8-52b7-45fb-9f31-5533cc2c85e8-iptables-alerter-script\") pod \"iptables-alerter-z2vvf\" (UID: \"c1b1a6b8-52b7-45fb-9f31-5533cc2c85e8\") " pod="openshift-network-operator/iptables-alerter-z2vvf"
Sep 17 17:56:54.285491 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.284810    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-multus-socket-dir-parent\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.285491 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.284854    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-multus-daemon-config\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.285491 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.284887    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7848"
Sep 17 17:56:54.285491 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.284888    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-log-socket\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.285491 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.285135    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0f02a062-ac1a-4caf-a488-3543ce36210f-sys-fs\") pod \"aws-ebs-csi-driver-node-plwn4\" (UID: \"0f02a062-ac1a-4caf-a488-3543ce36210f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plwn4"
Sep 17 17:56:54.285491 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:54.285134    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7848" podUID="a99a3614-e2dc-4d83-9bed-153654b4bafe"
Sep 17 17:56:54.285491 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.285171    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-hostroot\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.285491 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.285205    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-systemd-units\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.285491 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.285257    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/041688ba-3c1a-426b-84b3-ec168a6a8658-ovn-node-metrics-cert\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.285491 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.285324    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/83940065-1b61-4aca-819e-102e270dffc4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v6f7b\" (UID: \"83940065-1b61-4aca-819e-102e270dffc4\") " pod="openshift-multus/multus-additional-cni-plugins-v6f7b"
Sep 17 17:56:54.285491 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.285363    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5z8z\" (UniqueName: \"kubernetes.io/projected/83940065-1b61-4aca-819e-102e270dffc4-kube-api-access-k5z8z\") pod \"multus-additional-cni-plugins-v6f7b\" (UID: \"83940065-1b61-4aca-819e-102e270dffc4\") " pod="openshift-multus/multus-additional-cni-plugins-v6f7b"
Sep 17 17:56:54.285491 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.285391    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0f73bdd3-5b33-4d88-9d8b-eb540633a58e-agent-certs\") pod \"konnectivity-agent-8tpnw\" (UID: \"0f73bdd3-5b33-4d88-9d8b-eb540633a58e\") " pod="kube-system/konnectivity-agent-8tpnw"
Sep 17 17:56:54.285491 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.285425    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/83940065-1b61-4aca-819e-102e270dffc4-cni-binary-copy\") pod \"multus-additional-cni-plugins-v6f7b\" (UID: \"83940065-1b61-4aca-819e-102e270dffc4\") " pod="openshift-multus/multus-additional-cni-plugins-v6f7b"
Sep 17 17:56:54.285491 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.285462    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/83940065-1b61-4aca-819e-102e270dffc4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v6f7b\" (UID: \"83940065-1b61-4aca-819e-102e270dffc4\") " pod="openshift-multus/multus-additional-cni-plugins-v6f7b"
Sep 17 17:56:54.285491 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.285494    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0f02a062-ac1a-4caf-a488-3543ce36210f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-plwn4\" (UID: \"0f02a062-ac1a-4caf-a488-3543ce36210f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plwn4"
Sep 17 17:56:54.286323 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.285520    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-host-run-multus-certs\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.286323 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.285561    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-run-openvswitch\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.286323 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.285590    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0f02a062-ac1a-4caf-a488-3543ce36210f-device-dir\") pod \"aws-ebs-csi-driver-node-plwn4\" (UID: \"0f02a062-ac1a-4caf-a488-3543ce36210f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plwn4"
Sep 17 17:56:54.286323 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.285619    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1b1a6b8-52b7-45fb-9f31-5533cc2c85e8-host-slash\") pod \"iptables-alerter-z2vvf\" (UID: \"c1b1a6b8-52b7-45fb-9f31-5533cc2c85e8\") " pod="openshift-network-operator/iptables-alerter-z2vvf"
Sep 17 17:56:54.286323 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.285648    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-host-var-lib-cni-multus\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.286323 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.285673    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83940065-1b61-4aca-819e-102e270dffc4-system-cni-dir\") pod \"multus-additional-cni-plugins-v6f7b\" (UID: \"83940065-1b61-4aca-819e-102e270dffc4\") " pod="openshift-multus/multus-additional-cni-plugins-v6f7b"
Sep 17 17:56:54.286323 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.285707    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-host-run-netns\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.286323 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.285737    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-etc-kubernetes\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.286323 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.286055    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-host-run-ovn-kubernetes\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.287324 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.287302    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.290019 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.289999    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zsz9h"
Sep 17 17:56:54.290540 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.290520    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:56:54.290701 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.290681    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-knkbz\""
Sep 17 17:56:54.290860 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.290845    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 17:56:54.292290 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.292268    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:56:54.292366 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:54.292339    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xft5" podUID="33a2724c-596d-416e-99e2-583a9db84dcf"
Sep 17 17:56:54.293408 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.293388    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gxw8w\""
Sep 17 17:56:54.293470 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.293432    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 17:56:54.293576 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.293558    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 17:56:54.293749 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.293729    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 17:56:54.298366 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.298342    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:56:54.356435 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.356379    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal" event={"ID":"672f4fffca246081b3733694dc0095b6","Type":"ContainerStarted","Data":"3cb27b787512b26fdd4a920ed133697e2d0a1cdce51589dd00fc1ad3499a6a67"}
Sep 17 17:56:54.357671 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.357638    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" event={"ID":"d845ebff585bafdc97c5cabd0509365e","Type":"ContainerStarted","Data":"71ffd37ef4690c8b9f93403bf0af1a96c767767cad15ce7669f8bb748670b267"}
Sep 17 17:56:54.367388 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.367362    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-f554k"
Sep 17 17:56:54.373474 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.373452    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 17:56:54.383506 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.383482    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-f554k"
Sep 17 17:56:54.386293 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.386269    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr78c\" (UniqueName: \"kubernetes.io/projected/c1b1a6b8-52b7-45fb-9f31-5533cc2c85e8-kube-api-access-sr78c\") pod \"iptables-alerter-z2vvf\" (UID: \"c1b1a6b8-52b7-45fb-9f31-5533cc2c85e8\") " pod="openshift-network-operator/iptables-alerter-z2vvf"
Sep 17 17:56:54.386434 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.386298    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-var-lib-openvswitch\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.386434 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.386332    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gkh7\" (UniqueName: \"kubernetes.io/projected/4383fe04-b7e0-4dd9-b3cc-ddca3d57aed2-kube-api-access-6gkh7\") pod \"node-ca-zsz9h\" (UID: \"4383fe04-b7e0-4dd9-b3cc-ddca3d57aed2\") " pod="openshift-image-registry/node-ca-zsz9h"
Sep 17 17:56:54.386550 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.386417    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-var-lib-openvswitch\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.386550 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.386456    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-run-ovn\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.386550 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.386481    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-host-cni-bin\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.386550 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.386494    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-run-ovn\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.386550 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.386518    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xch9q\" (UniqueName: \"kubernetes.io/projected/041688ba-3c1a-426b-84b3-ec168a6a8658-kube-api-access-xch9q\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.386550 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.386547    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0f02a062-ac1a-4caf-a488-3543ce36210f-registration-dir\") pod \"aws-ebs-csi-driver-node-plwn4\" (UID: \"0f02a062-ac1a-4caf-a488-3543ce36210f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plwn4"
Sep 17 17:56:54.386793 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.386570    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-host-var-lib-kubelet\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.386793 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.386596    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-host-kubelet\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.386793 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.386623    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0f73bdd3-5b33-4d88-9d8b-eb540633a58e-konnectivity-ca\") pod \"konnectivity-agent-8tpnw\" (UID: \"0f73bdd3-5b33-4d88-9d8b-eb540633a58e\") " pod="kube-system/konnectivity-agent-8tpnw"
Sep 17 17:56:54.386793 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.386640    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-host-cni-bin\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.386793 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.386647    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0f02a062-ac1a-4caf-a488-3543ce36210f-registration-dir\") pod \"aws-ebs-csi-driver-node-plwn4\" (UID: \"0f02a062-ac1a-4caf-a488-3543ce36210f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plwn4"
Sep 17 17:56:54.386793 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.386687    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-host-kubelet\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.386793 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.386716    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-host-var-lib-kubelet\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.386793 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.386737    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c78hz\" (UniqueName: \"kubernetes.io/projected/87bce58a-35d3-497c-8d82-7f925fe82b76-kube-api-access-c78hz\") pod \"network-metrics-daemon-24pww\" (UID: \"87bce58a-35d3-497c-8d82-7f925fe82b76\") " pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:56:54.387122 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.386787    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-run-systemd\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.387122 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.386828    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/041688ba-3c1a-426b-84b3-ec168a6a8658-ovnkube-script-lib\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.387122 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.386850    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-run-systemd\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.387122 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.386855    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-host\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.387122 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.386915    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-tmp\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.387122 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.386937    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l7rn\" (UniqueName: \"kubernetes.io/projected/a99a3614-e2dc-4d83-9bed-153654b4bafe-kube-api-access-2l7rn\") pod \"network-check-target-z7848\" (UID: \"a99a3614-e2dc-4d83-9bed-153654b4bafe\") " pod="openshift-network-diagnostics/network-check-target-z7848"
Sep 17 17:56:54.387122 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.386957    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-system-cni-dir\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.387436 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.387209    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-system-cni-dir\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.387436 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.387257    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0f73bdd3-5b33-4d88-9d8b-eb540633a58e-konnectivity-ca\") pod \"konnectivity-agent-8tpnw\" (UID: \"0f73bdd3-5b33-4d88-9d8b-eb540633a58e\") " pod="kube-system/konnectivity-agent-8tpnw"
Sep 17 17:56:54.387518 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.387470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c1b1a6b8-52b7-45fb-9f31-5533cc2c85e8-iptables-alerter-script\") pod \"iptables-alerter-z2vvf\" (UID: \"c1b1a6b8-52b7-45fb-9f31-5533cc2c85e8\") " pod="openshift-network-operator/iptables-alerter-z2vvf"
Sep 17 17:56:54.387518 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.387484    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/041688ba-3c1a-426b-84b3-ec168a6a8658-ovnkube-script-lib\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.387518 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.387501    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-multus-socket-dir-parent\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.387673 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.387530    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-multus-daemon-config\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.387673 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.387556    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-log-socket\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.387673 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.387612    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-log-socket\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.387673 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.387636    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-multus-socket-dir-parent\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.387866 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.387678    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-etc-kubernetes\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.387866 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.387716    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-sys\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.387866 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.387749    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-lib-modules\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.387866 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.387779    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0f02a062-ac1a-4caf-a488-3543ce36210f-sys-fs\") pod \"aws-ebs-csi-driver-node-plwn4\" (UID: \"0f02a062-ac1a-4caf-a488-3543ce36210f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plwn4"
Sep 17 17:56:54.387866 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.387805    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-hostroot\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.387866 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.387824    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-systemd-units\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.388187 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.387879    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0f02a062-ac1a-4caf-a488-3543ce36210f-sys-fs\") pod \"aws-ebs-csi-driver-node-plwn4\" (UID: \"0f02a062-ac1a-4caf-a488-3543ce36210f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plwn4"
Sep 17 17:56:54.388187 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.387886    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-hostroot\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.388187 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.387901    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-systemd-units\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.388187 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.387928    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/041688ba-3c1a-426b-84b3-ec168a6a8658-ovn-node-metrics-cert\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.388187 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.387959    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/83940065-1b61-4aca-819e-102e270dffc4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v6f7b\" (UID: \"83940065-1b61-4aca-819e-102e270dffc4\") " pod="openshift-multus/multus-additional-cni-plugins-v6f7b"
Sep 17 17:56:54.388187 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.387981    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5z8z\" (UniqueName: \"kubernetes.io/projected/83940065-1b61-4aca-819e-102e270dffc4-kube-api-access-k5z8z\") pod \"multus-additional-cni-plugins-v6f7b\" (UID: \"83940065-1b61-4aca-819e-102e270dffc4\") " pod="openshift-multus/multus-additional-cni-plugins-v6f7b"
Sep 17 17:56:54.388187 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.388001    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0f73bdd3-5b33-4d88-9d8b-eb540633a58e-agent-certs\") pod \"konnectivity-agent-8tpnw\" (UID: \"0f73bdd3-5b33-4d88-9d8b-eb540633a58e\") " pod="kube-system/konnectivity-agent-8tpnw"
Sep 17 17:56:54.388187 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.388022    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c1b1a6b8-52b7-45fb-9f31-5533cc2c85e8-iptables-alerter-script\") pod \"iptables-alerter-z2vvf\" (UID: \"c1b1a6b8-52b7-45fb-9f31-5533cc2c85e8\") " pod="openshift-network-operator/iptables-alerter-z2vvf"
Sep 17 17:56:54.388187 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.388027    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/83940065-1b61-4aca-819e-102e270dffc4-cni-binary-copy\") pod \"multus-additional-cni-plugins-v6f7b\" (UID: \"83940065-1b61-4aca-819e-102e270dffc4\") " pod="openshift-multus/multus-additional-cni-plugins-v6f7b"
Sep 17 17:56:54.388187 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.388082    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/83940065-1b61-4aca-819e-102e270dffc4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v6f7b\" (UID: \"83940065-1b61-4aca-819e-102e270dffc4\") " pod="openshift-multus/multus-additional-cni-plugins-v6f7b"
Sep 17 17:56:54.388187 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.388125    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/33a2724c-596d-416e-99e2-583a9db84dcf-kubelet-config\") pod \"global-pull-secret-syncer-9xft5\" (UID: \"33a2724c-596d-416e-99e2-583a9db84dcf\") " pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:56:54.388187 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.388155    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-etc-modprobe-d\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.388187 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.388180    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-etc-tuned\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.388818 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.388213    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0f02a062-ac1a-4caf-a488-3543ce36210f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-plwn4\" (UID: \"0f02a062-ac1a-4caf-a488-3543ce36210f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plwn4"
Sep 17 17:56:54.388818 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.388240    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-host-run-multus-certs\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.388818 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.388272    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e73ac43a-8a55-40ea-902c-5b06dfcc5d0b-hosts-file\") pod \"node-resolver-qcqn9\" (UID: \"e73ac43a-8a55-40ea-902c-5b06dfcc5d0b\") " pod="openshift-dns/node-resolver-qcqn9"
Sep 17 17:56:54.388818 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.388295    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-etc-sysctl-d\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.388818 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.388319    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-run\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.388818 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.388592    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-run-openvswitch\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.388818 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.388604    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/83940065-1b61-4aca-819e-102e270dffc4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v6f7b\" (UID: \"83940065-1b61-4aca-819e-102e270dffc4\") " pod="openshift-multus/multus-additional-cni-plugins-v6f7b"
Sep 17 17:56:54.388818 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.388629    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4383fe04-b7e0-4dd9-b3cc-ddca3d57aed2-host\") pod \"node-ca-zsz9h\" (UID: \"4383fe04-b7e0-4dd9-b3cc-ddca3d57aed2\") " pod="openshift-image-registry/node-ca-zsz9h"
Sep 17 17:56:54.388818 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.388605    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-multus-daemon-config\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.388818 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.388659    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj8rl\" (UniqueName: \"kubernetes.io/projected/e73ac43a-8a55-40ea-902c-5b06dfcc5d0b-kube-api-access-sj8rl\") pod \"node-resolver-qcqn9\" (UID: \"e73ac43a-8a55-40ea-902c-5b06dfcc5d0b\") " pod="openshift-dns/node-resolver-qcqn9"
Sep 17 17:56:54.388818 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.388683    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0f02a062-ac1a-4caf-a488-3543ce36210f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-plwn4\" (UID: \"0f02a062-ac1a-4caf-a488-3543ce36210f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plwn4"
Sep 17 17:56:54.388818 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.388689    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-etc-sysconfig\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.388818 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.388728    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-etc-systemd\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.388818 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.388749    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-run-openvswitch\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.388818 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.388717    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-host-run-multus-certs\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.388818 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.388781    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0f02a062-ac1a-4caf-a488-3543ce36210f-device-dir\") pod \"aws-ebs-csi-driver-node-plwn4\" (UID: \"0f02a062-ac1a-4caf-a488-3543ce36210f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plwn4"
Sep 17 17:56:54.388818 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.388817    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/83940065-1b61-4aca-819e-102e270dffc4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v6f7b\" (UID: \"83940065-1b61-4aca-819e-102e270dffc4\") " pod="openshift-multus/multus-additional-cni-plugins-v6f7b"
Sep 17 17:56:54.389565 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.388827    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0f02a062-ac1a-4caf-a488-3543ce36210f-device-dir\") pod \"aws-ebs-csi-driver-node-plwn4\" (UID: \"0f02a062-ac1a-4caf-a488-3543ce36210f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plwn4"
Sep 17 17:56:54.389565 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.388819    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1b1a6b8-52b7-45fb-9f31-5533cc2c85e8-host-slash\") pod \"iptables-alerter-z2vvf\" (UID: \"c1b1a6b8-52b7-45fb-9f31-5533cc2c85e8\") " pod="openshift-network-operator/iptables-alerter-z2vvf"
Sep 17 17:56:54.389565 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.388840    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 17:56:54.389565 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.388871    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-host-var-lib-cni-multus\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.389565 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.388889    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1b1a6b8-52b7-45fb-9f31-5533cc2c85e8-host-slash\") pod \"iptables-alerter-z2vvf\" (UID: \"c1b1a6b8-52b7-45fb-9f31-5533cc2c85e8\") " pod="openshift-network-operator/iptables-alerter-z2vvf"
Sep 17 17:56:54.389565 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.388917    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83940065-1b61-4aca-819e-102e270dffc4-system-cni-dir\") pod \"multus-additional-cni-plugins-v6f7b\" (UID: \"83940065-1b61-4aca-819e-102e270dffc4\") " pod="openshift-multus/multus-additional-cni-plugins-v6f7b"
Sep 17 17:56:54.389565 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.388992    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83940065-1b61-4aca-819e-102e270dffc4-system-cni-dir\") pod \"multus-additional-cni-plugins-v6f7b\" (UID: \"83940065-1b61-4aca-819e-102e270dffc4\") " pod="openshift-multus/multus-additional-cni-plugins-v6f7b"
Sep 17 17:56:54.389565 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.388990    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/33a2724c-596d-416e-99e2-583a9db84dcf-dbus\") pod \"global-pull-secret-syncer-9xft5\" (UID: \"33a2724c-596d-416e-99e2-583a9db84dcf\") " pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:56:54.389565 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.389022    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/83940065-1b61-4aca-819e-102e270dffc4-cni-binary-copy\") pod \"multus-additional-cni-plugins-v6f7b\" (UID: \"83940065-1b61-4aca-819e-102e270dffc4\") " pod="openshift-multus/multus-additional-cni-plugins-v6f7b"
Sep 17 17:56:54.389565 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.389027    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/33a2724c-596d-416e-99e2-583a9db84dcf-original-pull-secret\") pod \"global-pull-secret-syncer-9xft5\" (UID: \"33a2724c-596d-416e-99e2-583a9db84dcf\") " pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:56:54.389565 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.389056    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-host-run-netns\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.389565 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.389039    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-host-var-lib-cni-multus\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.389565 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.389106    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-etc-kubernetes\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.389565 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.389160    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-host-run-ovn-kubernetes\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.389565 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.389228    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-etc-kubernetes\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.389565 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.389299    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-host-run-netns\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.389565 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.389314    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-host-run-ovn-kubernetes\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.389565 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.389312    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/041688ba-3c1a-426b-84b3-ec168a6a8658-ovnkube-config\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.390499 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.389434    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-var-lib-kubelet\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.390499 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.389490    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-cnibin\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.390499 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.389524    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-cni-binary-copy\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.390499 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.389553    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-host-slash\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.390499 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.389578    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-host-run-netns\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.390499 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.389604    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/83940065-1b61-4aca-819e-102e270dffc4-cnibin\") pod \"multus-additional-cni-plugins-v6f7b\" (UID: \"83940065-1b61-4aca-819e-102e270dffc4\") " pod="openshift-multus/multus-additional-cni-plugins-v6f7b"
Sep 17 17:56:54.390499 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.389635    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-etc-sysctl-conf\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.390499 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.389667    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s68kf\" (UniqueName: \"kubernetes.io/projected/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-kube-api-access-s68kf\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.390499 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.389694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-node-log\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.390499 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.389722    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/83940065-1b61-4aca-819e-102e270dffc4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v6f7b\" (UID: \"83940065-1b61-4aca-819e-102e270dffc4\") " pod="openshift-multus/multus-additional-cni-plugins-v6f7b"
Sep 17 17:56:54.390499 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.389754    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-host-var-lib-cni-bin\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.390499 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.389752    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-cnibin\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.390499 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.389764    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/83940065-1b61-4aca-819e-102e270dffc4-cnibin\") pod \"multus-additional-cni-plugins-v6f7b\" (UID: \"83940065-1b61-4aca-819e-102e270dffc4\") " pod="openshift-multus/multus-additional-cni-plugins-v6f7b"
Sep 17 17:56:54.390499 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.389806    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.390499 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.389846    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4383fe04-b7e0-4dd9-b3cc-ddca3d57aed2-serviceca\") pod \"node-ca-zsz9h\" (UID: \"4383fe04-b7e0-4dd9-b3cc-ddca3d57aed2\") " pod="openshift-image-registry/node-ca-zsz9h"
Sep 17 17:56:54.390499 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.389878    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0f02a062-ac1a-4caf-a488-3543ce36210f-etc-selinux\") pod \"aws-ebs-csi-driver-node-plwn4\" (UID: \"0f02a062-ac1a-4caf-a488-3543ce36210f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plwn4"
Sep 17 17:56:54.390499 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.389887    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/041688ba-3c1a-426b-84b3-ec168a6a8658-ovnkube-config\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.391216 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.389965    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/83940065-1b61-4aca-819e-102e270dffc4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v6f7b\" (UID: \"83940065-1b61-4aca-819e-102e270dffc4\") " pod="openshift-multus/multus-additional-cni-plugins-v6f7b"
Sep 17 17:56:54.391216 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.389976    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-host-run-k8s-cni-cncf-io\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.391216 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.389975    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-host-var-lib-cni-bin\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.391216 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.389910    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-host-run-k8s-cni-cncf-io\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.391216 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.389693    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-host-slash\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.391216 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.390032    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-node-log\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.391216 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.390038    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.391216 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.390044    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-multus-conf-dir\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.391216 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.390078    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-multus-conf-dir\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.391216 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.390094    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chzm2\" (UniqueName: \"kubernetes.io/projected/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-kube-api-access-chzm2\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.391216 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.390059    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0f02a062-ac1a-4caf-a488-3543ce36210f-etc-selinux\") pod \"aws-ebs-csi-driver-node-plwn4\" (UID: \"0f02a062-ac1a-4caf-a488-3543ce36210f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plwn4"
Sep 17 17:56:54.391216 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.390130    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-multus-cni-dir\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.391216 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.390157    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-etc-openvswitch\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.391216 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.390160    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-host-run-netns\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.391216 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.390197    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-host-cni-netd\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.391216 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.390221    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/041688ba-3c1a-426b-84b3-ec168a6a8658-env-overrides\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.391216 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.390239    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-etc-openvswitch\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.391854 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.390257    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/041688ba-3c1a-426b-84b3-ec168a6a8658-host-cni-netd\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.391854 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.390256    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-multus-cni-dir\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.391854 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.390259    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87bce58a-35d3-497c-8d82-7f925fe82b76-metrics-certs\") pod \"network-metrics-daemon-24pww\" (UID: \"87bce58a-35d3-497c-8d82-7f925fe82b76\") " pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:56:54.391854 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.390317    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e73ac43a-8a55-40ea-902c-5b06dfcc5d0b-tmp-dir\") pod \"node-resolver-qcqn9\" (UID: \"e73ac43a-8a55-40ea-902c-5b06dfcc5d0b\") " pod="openshift-dns/node-resolver-qcqn9"
Sep 17 17:56:54.391854 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.390370    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0f02a062-ac1a-4caf-a488-3543ce36210f-socket-dir\") pod \"aws-ebs-csi-driver-node-plwn4\" (UID: \"0f02a062-ac1a-4caf-a488-3543ce36210f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plwn4"
Sep 17 17:56:54.391854 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.390409    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qd97\" (UniqueName: \"kubernetes.io/projected/0f02a062-ac1a-4caf-a488-3543ce36210f-kube-api-access-6qd97\") pod \"aws-ebs-csi-driver-node-plwn4\" (UID: \"0f02a062-ac1a-4caf-a488-3543ce36210f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plwn4"
Sep 17 17:56:54.391854 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.390468    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-cni-binary-copy\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.391854 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.390529    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0f02a062-ac1a-4caf-a488-3543ce36210f-socket-dir\") pod \"aws-ebs-csi-driver-node-plwn4\" (UID: \"0f02a062-ac1a-4caf-a488-3543ce36210f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plwn4"
Sep 17 17:56:54.391854 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.390649    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/041688ba-3c1a-426b-84b3-ec168a6a8658-env-overrides\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.393016 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.392991    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/041688ba-3c1a-426b-84b3-ec168a6a8658-ovn-node-metrics-cert\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.393363 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.393327    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0f73bdd3-5b33-4d88-9d8b-eb540633a58e-agent-certs\") pod \"konnectivity-agent-8tpnw\" (UID: \"0f73bdd3-5b33-4d88-9d8b-eb540633a58e\") " pod="kube-system/konnectivity-agent-8tpnw"
Sep 17 17:56:54.400576 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.400555    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr78c\" (UniqueName: \"kubernetes.io/projected/c1b1a6b8-52b7-45fb-9f31-5533cc2c85e8-kube-api-access-sr78c\") pod \"iptables-alerter-z2vvf\" (UID: \"c1b1a6b8-52b7-45fb-9f31-5533cc2c85e8\") " pod="openshift-network-operator/iptables-alerter-z2vvf"
Sep 17 17:56:54.400698 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.400671    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5z8z\" (UniqueName: \"kubernetes.io/projected/83940065-1b61-4aca-819e-102e270dffc4-kube-api-access-k5z8z\") pod \"multus-additional-cni-plugins-v6f7b\" (UID: \"83940065-1b61-4aca-819e-102e270dffc4\") " pod="openshift-multus/multus-additional-cni-plugins-v6f7b"
Sep 17 17:56:54.400853 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.400826    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xch9q\" (UniqueName: \"kubernetes.io/projected/041688ba-3c1a-426b-84b3-ec168a6a8658-kube-api-access-xch9q\") pod \"ovnkube-node-bz9ng\" (UID: \"041688ba-3c1a-426b-84b3-ec168a6a8658\") " pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.406147 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.406103    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qd97\" (UniqueName: \"kubernetes.io/projected/0f02a062-ac1a-4caf-a488-3543ce36210f-kube-api-access-6qd97\") pod \"aws-ebs-csi-driver-node-plwn4\" (UID: \"0f02a062-ac1a-4caf-a488-3543ce36210f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plwn4"
Sep 17 17:56:54.412243 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.412215    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s68kf\" (UniqueName: \"kubernetes.io/projected/e0c61526-ed8d-40e7-b87e-e9cb3ee45221-kube-api-access-s68kf\") pod \"multus-4nrxj\" (UID: \"e0c61526-ed8d-40e7-b87e-e9cb3ee45221\") " pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.491915 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.491810    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c78hz\" (UniqueName: \"kubernetes.io/projected/87bce58a-35d3-497c-8d82-7f925fe82b76-kube-api-access-c78hz\") pod \"network-metrics-daemon-24pww\" (UID: \"87bce58a-35d3-497c-8d82-7f925fe82b76\") " pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:56:54.491915 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.491861    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-host\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.491915 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.491887    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-tmp\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.492162 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.491988    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-host\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.492162 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492020    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l7rn\" (UniqueName: \"kubernetes.io/projected/a99a3614-e2dc-4d83-9bed-153654b4bafe-kube-api-access-2l7rn\") pod \"network-check-target-z7848\" (UID: \"a99a3614-e2dc-4d83-9bed-153654b4bafe\") " pod="openshift-network-diagnostics/network-check-target-z7848"
Sep 17 17:56:54.492162 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492055    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-etc-kubernetes\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.492162 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492078    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-sys\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.492162 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492101    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-lib-modules\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.492162 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492134    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/33a2724c-596d-416e-99e2-583a9db84dcf-kubelet-config\") pod \"global-pull-secret-syncer-9xft5\" (UID: \"33a2724c-596d-416e-99e2-583a9db84dcf\") " pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:56:54.492162 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492158    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-etc-modprobe-d\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.492394 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492168    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-etc-kubernetes\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.492394 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492183    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-etc-tuned\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.492394 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492281    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-lib-modules\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.492394 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492310    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e73ac43a-8a55-40ea-902c-5b06dfcc5d0b-hosts-file\") pod \"node-resolver-qcqn9\" (UID: \"e73ac43a-8a55-40ea-902c-5b06dfcc5d0b\") " pod="openshift-dns/node-resolver-qcqn9"
Sep 17 17:56:54.492394 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492338    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-etc-sysctl-d\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.492394 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492350    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/33a2724c-596d-416e-99e2-583a9db84dcf-kubelet-config\") pod \"global-pull-secret-syncer-9xft5\" (UID: \"33a2724c-596d-416e-99e2-583a9db84dcf\") " pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:56:54.492394 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492364    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-run\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.492394 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492367    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-sys\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.492693 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492422    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-run\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.492693 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492428    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e73ac43a-8a55-40ea-902c-5b06dfcc5d0b-hosts-file\") pod \"node-resolver-qcqn9\" (UID: \"e73ac43a-8a55-40ea-902c-5b06dfcc5d0b\") " pod="openshift-dns/node-resolver-qcqn9"
Sep 17 17:56:54.492693 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492466    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-etc-sysctl-d\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.492693 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492475    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4383fe04-b7e0-4dd9-b3cc-ddca3d57aed2-host\") pod \"node-ca-zsz9h\" (UID: \"4383fe04-b7e0-4dd9-b3cc-ddca3d57aed2\") " pod="openshift-image-registry/node-ca-zsz9h"
Sep 17 17:56:54.492693 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492477    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-etc-modprobe-d\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.492693 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492506    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj8rl\" (UniqueName: \"kubernetes.io/projected/e73ac43a-8a55-40ea-902c-5b06dfcc5d0b-kube-api-access-sj8rl\") pod \"node-resolver-qcqn9\" (UID: \"e73ac43a-8a55-40ea-902c-5b06dfcc5d0b\") " pod="openshift-dns/node-resolver-qcqn9"
Sep 17 17:56:54.492693 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492511    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4383fe04-b7e0-4dd9-b3cc-ddca3d57aed2-host\") pod \"node-ca-zsz9h\" (UID: \"4383fe04-b7e0-4dd9-b3cc-ddca3d57aed2\") " pod="openshift-image-registry/node-ca-zsz9h"
Sep 17 17:56:54.492693 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492537    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-etc-sysconfig\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.492693 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492558    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-etc-systemd\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.492693 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492580    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/33a2724c-596d-416e-99e2-583a9db84dcf-dbus\") pod \"global-pull-secret-syncer-9xft5\" (UID: \"33a2724c-596d-416e-99e2-583a9db84dcf\") " pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:56:54.492693 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492587    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-etc-sysconfig\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.492693 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492597    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/33a2724c-596d-416e-99e2-583a9db84dcf-original-pull-secret\") pod \"global-pull-secret-syncer-9xft5\" (UID: \"33a2724c-596d-416e-99e2-583a9db84dcf\") " pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:56:54.492693 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492616    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-var-lib-kubelet\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.492693 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492633    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-etc-systemd\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.492693 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492644    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-etc-sysctl-conf\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.492693 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492674    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4383fe04-b7e0-4dd9-b3cc-ddca3d57aed2-serviceca\") pod \"node-ca-zsz9h\" (UID: \"4383fe04-b7e0-4dd9-b3cc-ddca3d57aed2\") " pod="openshift-image-registry/node-ca-zsz9h"
Sep 17 17:56:54.492693 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492683    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/33a2724c-596d-416e-99e2-583a9db84dcf-dbus\") pod \"global-pull-secret-syncer-9xft5\" (UID: \"33a2724c-596d-416e-99e2-583a9db84dcf\") " pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:56:54.493466 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492704    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chzm2\" (UniqueName: \"kubernetes.io/projected/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-kube-api-access-chzm2\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.493466 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:54.492722    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:54.493466 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492732    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87bce58a-35d3-497c-8d82-7f925fe82b76-metrics-certs\") pod \"network-metrics-daemon-24pww\" (UID: \"87bce58a-35d3-497c-8d82-7f925fe82b76\") " pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:56:54.493466 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492753    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e73ac43a-8a55-40ea-902c-5b06dfcc5d0b-tmp-dir\") pod \"node-resolver-qcqn9\" (UID: \"e73ac43a-8a55-40ea-902c-5b06dfcc5d0b\") " pod="openshift-dns/node-resolver-qcqn9"
Sep 17 17:56:54.493466 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492763    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-var-lib-kubelet\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.493466 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492776    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gkh7\" (UniqueName: \"kubernetes.io/projected/4383fe04-b7e0-4dd9-b3cc-ddca3d57aed2-kube-api-access-6gkh7\") pod \"node-ca-zsz9h\" (UID: \"4383fe04-b7e0-4dd9-b3cc-ddca3d57aed2\") " pod="openshift-image-registry/node-ca-zsz9h"
Sep 17 17:56:54.493466 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:54.492809    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33a2724c-596d-416e-99e2-583a9db84dcf-original-pull-secret podName:33a2724c-596d-416e-99e2-583a9db84dcf nodeName:}" failed. No retries permitted until 2026-09-17 17:56:54.992785132 +0000 UTC m=+3.083406776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/33a2724c-596d-416e-99e2-583a9db84dcf-original-pull-secret") pod "global-pull-secret-syncer-9xft5" (UID: "33a2724c-596d-416e-99e2-583a9db84dcf") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:54.493466 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.492825    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-etc-sysctl-conf\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.493466 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:54.492893    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:54.493466 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:54.493167    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87bce58a-35d3-497c-8d82-7f925fe82b76-metrics-certs podName:87bce58a-35d3-497c-8d82-7f925fe82b76 nodeName:}" failed. No retries permitted until 2026-09-17 17:56:54.993148877 +0000 UTC m=+3.083770510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87bce58a-35d3-497c-8d82-7f925fe82b76-metrics-certs") pod "network-metrics-daemon-24pww" (UID: "87bce58a-35d3-497c-8d82-7f925fe82b76") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:54.493466 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.493210    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4383fe04-b7e0-4dd9-b3cc-ddca3d57aed2-serviceca\") pod \"node-ca-zsz9h\" (UID: \"4383fe04-b7e0-4dd9-b3cc-ddca3d57aed2\") " pod="openshift-image-registry/node-ca-zsz9h"
Sep 17 17:56:54.493466 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.493226    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e73ac43a-8a55-40ea-902c-5b06dfcc5d0b-tmp-dir\") pod \"node-resolver-qcqn9\" (UID: \"e73ac43a-8a55-40ea-902c-5b06dfcc5d0b\") " pod="openshift-dns/node-resolver-qcqn9"
Sep 17 17:56:54.495093 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.495063    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-etc-tuned\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.495200 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.495116    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-tmp\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.503560 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:54.503537    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:56:54.503560 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:54.503563    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:56:54.503721 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:54.503578    2581 projected.go:196] Error preparing data for projected volume kube-api-access-2l7rn for pod openshift-network-diagnostics/network-check-target-z7848: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:54.503721 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:54.503654    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a99a3614-e2dc-4d83-9bed-153654b4bafe-kube-api-access-2l7rn podName:a99a3614-e2dc-4d83-9bed-153654b4bafe nodeName:}" failed. No retries permitted until 2026-09-17 17:56:55.003636028 +0000 UTC m=+3.094257660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2l7rn" (UniqueName: "kubernetes.io/projected/a99a3614-e2dc-4d83-9bed-153654b4bafe-kube-api-access-2l7rn") pod "network-check-target-z7848" (UID: "a99a3614-e2dc-4d83-9bed-153654b4bafe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:54.507039 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.507017    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c78hz\" (UniqueName: \"kubernetes.io/projected/87bce58a-35d3-497c-8d82-7f925fe82b76-kube-api-access-c78hz\") pod \"network-metrics-daemon-24pww\" (UID: \"87bce58a-35d3-497c-8d82-7f925fe82b76\") " pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:56:54.507303 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.507283    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj8rl\" (UniqueName: \"kubernetes.io/projected/e73ac43a-8a55-40ea-902c-5b06dfcc5d0b-kube-api-access-sj8rl\") pod \"node-resolver-qcqn9\" (UID: \"e73ac43a-8a55-40ea-902c-5b06dfcc5d0b\") " pod="openshift-dns/node-resolver-qcqn9"
Sep 17 17:56:54.509547 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.509528    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chzm2\" (UniqueName: \"kubernetes.io/projected/f11a6327-77aa-444a-a4a4-abdddd8b9ec3-kube-api-access-chzm2\") pod \"tuned-2c98r\" (UID: \"f11a6327-77aa-444a-a4a4-abdddd8b9ec3\") " pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.509751 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.509731    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gkh7\" (UniqueName: \"kubernetes.io/projected/4383fe04-b7e0-4dd9-b3cc-ddca3d57aed2-kube-api-access-6gkh7\") pod \"node-ca-zsz9h\" (UID: \"4383fe04-b7e0-4dd9-b3cc-ddca3d57aed2\") " pod="openshift-image-registry/node-ca-zsz9h"
Sep 17 17:56:54.575168 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.575130    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z2vvf"
Sep 17 17:56:54.588268 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.588227    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4nrxj"
Sep 17 17:56:54.597400 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.597374    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:56:54.603500 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.603476    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v6f7b"
Sep 17 17:56:54.611409 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.611385    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8tpnw"
Sep 17 17:56:54.619480 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.619457    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plwn4"
Sep 17 17:56:54.625764 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.625744    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qcqn9"
Sep 17 17:56:54.632649 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.632629    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2c98r"
Sep 17 17:56:54.638235 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.638216    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zsz9h"
Sep 17 17:56:54.998193 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.998159    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87bce58a-35d3-497c-8d82-7f925fe82b76-metrics-certs\") pod \"network-metrics-daemon-24pww\" (UID: \"87bce58a-35d3-497c-8d82-7f925fe82b76\") " pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:56:54.998413 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:54.998219    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/33a2724c-596d-416e-99e2-583a9db84dcf-original-pull-secret\") pod \"global-pull-secret-syncer-9xft5\" (UID: \"33a2724c-596d-416e-99e2-583a9db84dcf\") " pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:56:54.998413 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:54.998318    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:54.998413 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:54.998334    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:54.998413 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:54.998379    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33a2724c-596d-416e-99e2-583a9db84dcf-original-pull-secret podName:33a2724c-596d-416e-99e2-583a9db84dcf nodeName:}" failed. No retries permitted until 2026-09-17 17:56:55.998362015 +0000 UTC m=+4.088983646 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/33a2724c-596d-416e-99e2-583a9db84dcf-original-pull-secret") pod "global-pull-secret-syncer-9xft5" (UID: "33a2724c-596d-416e-99e2-583a9db84dcf") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:54.998413 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:54.998393    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87bce58a-35d3-497c-8d82-7f925fe82b76-metrics-certs podName:87bce58a-35d3-497c-8d82-7f925fe82b76 nodeName:}" failed. No retries permitted until 2026-09-17 17:56:55.998386638 +0000 UTC m=+4.089008269 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87bce58a-35d3-497c-8d82-7f925fe82b76-metrics-certs") pod "network-metrics-daemon-24pww" (UID: "87bce58a-35d3-497c-8d82-7f925fe82b76") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:55.098855 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:55.098811    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l7rn\" (UniqueName: \"kubernetes.io/projected/a99a3614-e2dc-4d83-9bed-153654b4bafe-kube-api-access-2l7rn\") pod \"network-check-target-z7848\" (UID: \"a99a3614-e2dc-4d83-9bed-153654b4bafe\") " pod="openshift-network-diagnostics/network-check-target-z7848"
Sep 17 17:56:55.099039 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:55.099007    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:56:55.099039 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:55.099034    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:56:55.099128 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:55.099045    2581 projected.go:196] Error preparing data for projected volume kube-api-access-2l7rn for pod openshift-network-diagnostics/network-check-target-z7848: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:55.099128 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:55.099117    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a99a3614-e2dc-4d83-9bed-153654b4bafe-kube-api-access-2l7rn podName:a99a3614-e2dc-4d83-9bed-153654b4bafe nodeName:}" failed. No retries permitted until 2026-09-17 17:56:56.099095474 +0000 UTC m=+4.189717135 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2l7rn" (UniqueName: "kubernetes.io/projected/a99a3614-e2dc-4d83-9bed-153654b4bafe-kube-api-access-2l7rn") pod "network-check-target-z7848" (UID: "a99a3614-e2dc-4d83-9bed-153654b4bafe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:55.349938 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:55.349881    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode73ac43a_8a55_40ea_902c_5b06dfcc5d0b.slice/crio-f5a0775384d18fb150da2cc1fbedfb613de446a9bc680e2fae7a7ea58a3c72cb WatchSource:0}: Error finding container f5a0775384d18fb150da2cc1fbedfb613de446a9bc680e2fae7a7ea58a3c72cb: Status 404 returned error can't find the container with id f5a0775384d18fb150da2cc1fbedfb613de446a9bc680e2fae7a7ea58a3c72cb
Sep 17 17:56:55.351104 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:55.351072    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c61526_ed8d_40e7_b87e_e9cb3ee45221.slice/crio-65aedf7edfa8b66a472ff1c458ab652f0ec529e79db7832067a3453bcead9fde WatchSource:0}: Error finding container 65aedf7edfa8b66a472ff1c458ab652f0ec529e79db7832067a3453bcead9fde: Status 404 returned error can't find the container with id 65aedf7edfa8b66a472ff1c458ab652f0ec529e79db7832067a3453bcead9fde
Sep 17 17:56:55.351804 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:55.351781    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:56:55.352097 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:55.351909    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xft5" podUID="33a2724c-596d-416e-99e2-583a9db84dcf"
Sep 17 17:56:55.352361 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:55.352332    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b1a6b8_52b7_45fb_9f31_5533cc2c85e8.slice/crio-1d5891e3150fe37702c18a012a59ab8df97b0ceca99139321f874b721ac83334 WatchSource:0}: Error finding container 1d5891e3150fe37702c18a012a59ab8df97b0ceca99139321f874b721ac83334: Status 404 returned error can't find the container with id 1d5891e3150fe37702c18a012a59ab8df97b0ceca99139321f874b721ac83334
Sep 17 17:56:55.354143 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:55.354120    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf11a6327_77aa_444a_a4a4_abdddd8b9ec3.slice/crio-8679dd489d768a35f77afdfd302b522bb3659fa5ce71a2ea63d162a8735f3709 WatchSource:0}: Error finding container 8679dd489d768a35f77afdfd302b522bb3659fa5ce71a2ea63d162a8735f3709: Status 404 returned error can't find the container with id 8679dd489d768a35f77afdfd302b522bb3659fa5ce71a2ea63d162a8735f3709
Sep 17 17:56:55.355785 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:55.355760    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4383fe04_b7e0_4dd9_b3cc_ddca3d57aed2.slice/crio-396eb990eccec0d9da05011976e43e38b04752597596ae5be9c31a1298c0794c WatchSource:0}: Error finding container 396eb990eccec0d9da05011976e43e38b04752597596ae5be9c31a1298c0794c: Status 404 returned error can't find the container with id 396eb990eccec0d9da05011976e43e38b04752597596ae5be9c31a1298c0794c
Sep 17 17:56:55.360142 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:55.360116    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4nrxj" event={"ID":"e0c61526-ed8d-40e7-b87e-e9cb3ee45221","Type":"ContainerStarted","Data":"65aedf7edfa8b66a472ff1c458ab652f0ec529e79db7832067a3453bcead9fde"}
Sep 17 17:56:55.361442 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:55.361411    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qcqn9" event={"ID":"e73ac43a-8a55-40ea-902c-5b06dfcc5d0b","Type":"ContainerStarted","Data":"f5a0775384d18fb150da2cc1fbedfb613de446a9bc680e2fae7a7ea58a3c72cb"}
Sep 17 17:56:55.362571 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:55.362544    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zsz9h" event={"ID":"4383fe04-b7e0-4dd9-b3cc-ddca3d57aed2","Type":"ContainerStarted","Data":"396eb990eccec0d9da05011976e43e38b04752597596ae5be9c31a1298c0794c"}
Sep 17 17:56:55.363691 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:55.363672    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2c98r" event={"ID":"f11a6327-77aa-444a-a4a4-abdddd8b9ec3","Type":"ContainerStarted","Data":"8679dd489d768a35f77afdfd302b522bb3659fa5ce71a2ea63d162a8735f3709"}
Sep 17 17:56:55.364675 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:55.364652    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z2vvf" event={"ID":"c1b1a6b8-52b7-45fb-9f31-5533cc2c85e8","Type":"ContainerStarted","Data":"1d5891e3150fe37702c18a012a59ab8df97b0ceca99139321f874b721ac83334"}
Sep 17 17:56:55.379537 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:55.379511    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f02a062_ac1a_4caf_a488_3543ce36210f.slice/crio-039140fa356d53c0c00a4417d020d408206950ce5e04ddb440d62b0d644541af WatchSource:0}: Error finding container 039140fa356d53c0c00a4417d020d408206950ce5e04ddb440d62b0d644541af: Status 404 returned error can't find the container with id 039140fa356d53c0c00a4417d020d408206950ce5e04ddb440d62b0d644541af
Sep 17 17:56:55.380752 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:55.380728    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod041688ba_3c1a_426b_84b3_ec168a6a8658.slice/crio-0dedca7f0e50a753cd11c24eb41c16c008bf076e6de99206096c43f4e1c35024 WatchSource:0}: Error finding container 0dedca7f0e50a753cd11c24eb41c16c008bf076e6de99206096c43f4e1c35024: Status 404 returned error can't find the container with id 0dedca7f0e50a753cd11c24eb41c16c008bf076e6de99206096c43f4e1c35024
Sep 17 17:56:55.382453 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:56:55.382431    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f73bdd3_5b33_4d88_9d8b_eb540633a58e.slice/crio-2417584ee5e93c94a6ab14955e7c98987151d5f00d06093ea685b85b627d4de3 WatchSource:0}: Error finding container 2417584ee5e93c94a6ab14955e7c98987151d5f00d06093ea685b85b627d4de3: Status 404 returned error can't find the container with id 2417584ee5e93c94a6ab14955e7c98987151d5f00d06093ea685b85b627d4de3
Sep 17 17:56:55.386703 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:55.386563    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:51:54 +0000 UTC" deadline="2028-02-17 13:59:37.710802963 +0000 UTC"
Sep 17 17:56:55.386703 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:55.386627    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12428h2m42.324199021s"
Sep 17 17:56:56.007361 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:56.007266    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/33a2724c-596d-416e-99e2-583a9db84dcf-original-pull-secret\") pod \"global-pull-secret-syncer-9xft5\" (UID: \"33a2724c-596d-416e-99e2-583a9db84dcf\") " pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:56:56.007361 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:56.007339    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87bce58a-35d3-497c-8d82-7f925fe82b76-metrics-certs\") pod \"network-metrics-daemon-24pww\" (UID: \"87bce58a-35d3-497c-8d82-7f925fe82b76\") " pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:56:56.007604 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:56.007491    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:56.007604 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:56.007557    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87bce58a-35d3-497c-8d82-7f925fe82b76-metrics-certs podName:87bce58a-35d3-497c-8d82-7f925fe82b76 nodeName:}" failed. No retries permitted until 2026-09-17 17:56:58.007537533 +0000 UTC m=+6.098159167 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87bce58a-35d3-497c-8d82-7f925fe82b76-metrics-certs") pod "network-metrics-daemon-24pww" (UID: "87bce58a-35d3-497c-8d82-7f925fe82b76") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:56.008312 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:56.008287    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:56.008509 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:56.008354    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33a2724c-596d-416e-99e2-583a9db84dcf-original-pull-secret podName:33a2724c-596d-416e-99e2-583a9db84dcf nodeName:}" failed. No retries permitted until 2026-09-17 17:56:58.008337673 +0000 UTC m=+6.098959318 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/33a2724c-596d-416e-99e2-583a9db84dcf-original-pull-secret") pod "global-pull-secret-syncer-9xft5" (UID: "33a2724c-596d-416e-99e2-583a9db84dcf") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:56.108435 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:56.108387    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l7rn\" (UniqueName: \"kubernetes.io/projected/a99a3614-e2dc-4d83-9bed-153654b4bafe-kube-api-access-2l7rn\") pod \"network-check-target-z7848\" (UID: \"a99a3614-e2dc-4d83-9bed-153654b4bafe\") " pod="openshift-network-diagnostics/network-check-target-z7848"
Sep 17 17:56:56.108631 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:56.108563    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:56:56.108631 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:56.108593    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:56:56.108631 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:56.108606    2581 projected.go:196] Error preparing data for projected volume kube-api-access-2l7rn for pod openshift-network-diagnostics/network-check-target-z7848: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:56.108775 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:56.108671    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a99a3614-e2dc-4d83-9bed-153654b4bafe-kube-api-access-2l7rn podName:a99a3614-e2dc-4d83-9bed-153654b4bafe nodeName:}" failed. No retries permitted until 2026-09-17 17:56:58.10865115 +0000 UTC m=+6.199272804 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2l7rn" (UniqueName: "kubernetes.io/projected/a99a3614-e2dc-4d83-9bed-153654b4bafe-kube-api-access-2l7rn") pod "network-check-target-z7848" (UID: "a99a3614-e2dc-4d83-9bed-153654b4bafe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:56.355755 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:56.355667    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7848"
Sep 17 17:56:56.356631 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:56.355812    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7848" podUID="a99a3614-e2dc-4d83-9bed-153654b4bafe"
Sep 17 17:56:56.356631 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:56.355897    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:56:56.356631 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:56.356091    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24pww" podUID="87bce58a-35d3-497c-8d82-7f925fe82b76"
Sep 17 17:56:56.386930 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:56.386853    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:51:54 +0000 UTC" deadline="2028-02-27 16:21:56.059378972 +0000 UTC"
Sep 17 17:56:56.386930 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:56.386890    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12670h24m59.672493738s"
Sep 17 17:56:56.395647 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:56.395599    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v6f7b" event={"ID":"83940065-1b61-4aca-819e-102e270dffc4","Type":"ContainerStarted","Data":"02fdd6fb388f096c219cd5b4b35340ee23941cee2139f85705917a7e22840c62"}
Sep 17 17:56:56.409224 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:56.409186    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8tpnw" event={"ID":"0f73bdd3-5b33-4d88-9d8b-eb540633a58e","Type":"ContainerStarted","Data":"2417584ee5e93c94a6ab14955e7c98987151d5f00d06093ea685b85b627d4de3"}
Sep 17 17:56:56.423051 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:56.423007    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal" event={"ID":"672f4fffca246081b3733694dc0095b6","Type":"ContainerStarted","Data":"47c046d08f49e432c02b63294bb72c821c6819f767e8eba2ec30f602e29375e5"}
Sep 17 17:56:56.437150 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:56.435313    2581 generic.go:356] "Generic (PLEG): container finished" podID="d845ebff585bafdc97c5cabd0509365e" containerID="5762cd36e7f72077b19c2cc89e0176a410a8f3f10873995c96e7891d9cf6bfd3" exitCode=0
Sep 17 17:56:56.437150 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:56.435408    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" event={"ID":"d845ebff585bafdc97c5cabd0509365e","Type":"ContainerDied","Data":"5762cd36e7f72077b19c2cc89e0176a410a8f3f10873995c96e7891d9cf6bfd3"}
Sep 17 17:56:56.441161 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:56.441031    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng" event={"ID":"041688ba-3c1a-426b-84b3-ec168a6a8658","Type":"ContainerStarted","Data":"0dedca7f0e50a753cd11c24eb41c16c008bf076e6de99206096c43f4e1c35024"}
Sep 17 17:56:56.444949 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:56.444890    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-28.ec2.internal" podStartSLOduration=2.444869855 podStartE2EDuration="2.444869855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:56:56.444656505 +0000 UTC m=+4.535278159" watchObservedRunningTime="2026-09-17 17:56:56.444869855 +0000 UTC m=+4.535491490"
Sep 17 17:56:56.453293 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:56.452975    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plwn4" event={"ID":"0f02a062-ac1a-4caf-a488-3543ce36210f","Type":"ContainerStarted","Data":"039140fa356d53c0c00a4417d020d408206950ce5e04ddb440d62b0d644541af"}
Sep 17 17:56:57.352978 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:57.352714    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:56:57.352978 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:57.352848    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xft5" podUID="33a2724c-596d-416e-99e2-583a9db84dcf"
Sep 17 17:56:57.488605 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:57.488570    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" event={"ID":"d845ebff585bafdc97c5cabd0509365e","Type":"ContainerStarted","Data":"619e392263614bef1e5c15ee432640a116a055955925c604d7800ec1cdd3fab3"}
Sep 17 17:56:58.024961 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:58.024921    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/33a2724c-596d-416e-99e2-583a9db84dcf-original-pull-secret\") pod \"global-pull-secret-syncer-9xft5\" (UID: \"33a2724c-596d-416e-99e2-583a9db84dcf\") " pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:56:58.025146 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:58.024985    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87bce58a-35d3-497c-8d82-7f925fe82b76-metrics-certs\") pod \"network-metrics-daemon-24pww\" (UID: \"87bce58a-35d3-497c-8d82-7f925fe82b76\") " pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:56:58.025146 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:58.025111    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:58.025282 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:58.025172    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87bce58a-35d3-497c-8d82-7f925fe82b76-metrics-certs podName:87bce58a-35d3-497c-8d82-7f925fe82b76 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:02.025153283 +0000 UTC m=+10.115774929 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87bce58a-35d3-497c-8d82-7f925fe82b76-metrics-certs") pod "network-metrics-daemon-24pww" (UID: "87bce58a-35d3-497c-8d82-7f925fe82b76") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:56:58.025855 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:58.025571    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:58.025855 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:58.025621    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33a2724c-596d-416e-99e2-583a9db84dcf-original-pull-secret podName:33a2724c-596d-416e-99e2-583a9db84dcf nodeName:}" failed. No retries permitted until 2026-09-17 17:57:02.025607458 +0000 UTC m=+10.116229103 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/33a2724c-596d-416e-99e2-583a9db84dcf-original-pull-secret") pod "global-pull-secret-syncer-9xft5" (UID: "33a2724c-596d-416e-99e2-583a9db84dcf") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:56:58.125281 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:58.125248    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l7rn\" (UniqueName: \"kubernetes.io/projected/a99a3614-e2dc-4d83-9bed-153654b4bafe-kube-api-access-2l7rn\") pod \"network-check-target-z7848\" (UID: \"a99a3614-e2dc-4d83-9bed-153654b4bafe\") " pod="openshift-network-diagnostics/network-check-target-z7848"
Sep 17 17:56:58.125467 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:58.125419    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:56:58.125467 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:58.125436    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:56:58.125467 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:58.125448    2581 projected.go:196] Error preparing data for projected volume kube-api-access-2l7rn for pod openshift-network-diagnostics/network-check-target-z7848: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:58.125624 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:58.125502    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a99a3614-e2dc-4d83-9bed-153654b4bafe-kube-api-access-2l7rn podName:a99a3614-e2dc-4d83-9bed-153654b4bafe nodeName:}" failed. No retries permitted until 2026-09-17 17:57:02.125484296 +0000 UTC m=+10.216105939 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2l7rn" (UniqueName: "kubernetes.io/projected/a99a3614-e2dc-4d83-9bed-153654b4bafe-kube-api-access-2l7rn") pod "network-check-target-z7848" (UID: "a99a3614-e2dc-4d83-9bed-153654b4bafe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:56:58.352115 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:58.352038    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:56:58.352273 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:58.352045    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7848"
Sep 17 17:56:58.352273 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:58.352192    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24pww" podUID="87bce58a-35d3-497c-8d82-7f925fe82b76"
Sep 17 17:56:58.352273 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:58.352244    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7848" podUID="a99a3614-e2dc-4d83-9bed-153654b4bafe"
Sep 17 17:56:59.352732 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:56:59.352480    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:56:59.352732 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:56:59.352599    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xft5" podUID="33a2724c-596d-416e-99e2-583a9db84dcf"
Sep 17 17:57:00.352079 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:00.352045    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7848"
Sep 17 17:57:00.352278 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:00.352182    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7848" podUID="a99a3614-e2dc-4d83-9bed-153654b4bafe"
Sep 17 17:57:00.352587 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:00.352565    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:57:00.352700 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:00.352674    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24pww" podUID="87bce58a-35d3-497c-8d82-7f925fe82b76"
Sep 17 17:57:01.352389 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:01.351831    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:57:01.352389 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:01.351995    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xft5" podUID="33a2724c-596d-416e-99e2-583a9db84dcf"
Sep 17 17:57:02.059536 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:02.059483    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/33a2724c-596d-416e-99e2-583a9db84dcf-original-pull-secret\") pod \"global-pull-secret-syncer-9xft5\" (UID: \"33a2724c-596d-416e-99e2-583a9db84dcf\") " pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:57:02.059728 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:02.059558    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87bce58a-35d3-497c-8d82-7f925fe82b76-metrics-certs\") pod \"network-metrics-daemon-24pww\" (UID: \"87bce58a-35d3-497c-8d82-7f925fe82b76\") " pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:57:02.059728 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:02.059690    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:02.059841 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:02.059758    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87bce58a-35d3-497c-8d82-7f925fe82b76-metrics-certs podName:87bce58a-35d3-497c-8d82-7f925fe82b76 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:10.059737021 +0000 UTC m=+18.150358650 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87bce58a-35d3-497c-8d82-7f925fe82b76-metrics-certs") pod "network-metrics-daemon-24pww" (UID: "87bce58a-35d3-497c-8d82-7f925fe82b76") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:02.060132 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:02.060025    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:02.060132 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:02.060105    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33a2724c-596d-416e-99e2-583a9db84dcf-original-pull-secret podName:33a2724c-596d-416e-99e2-583a9db84dcf nodeName:}" failed. No retries permitted until 2026-09-17 17:57:10.060088798 +0000 UTC m=+18.150710443 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/33a2724c-596d-416e-99e2-583a9db84dcf-original-pull-secret") pod "global-pull-secret-syncer-9xft5" (UID: "33a2724c-596d-416e-99e2-583a9db84dcf") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:02.160502 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:02.160390    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l7rn\" (UniqueName: \"kubernetes.io/projected/a99a3614-e2dc-4d83-9bed-153654b4bafe-kube-api-access-2l7rn\") pod \"network-check-target-z7848\" (UID: \"a99a3614-e2dc-4d83-9bed-153654b4bafe\") " pod="openshift-network-diagnostics/network-check-target-z7848"
Sep 17 17:57:02.160684 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:02.160603    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:57:02.160684 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:02.160632    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:57:02.160684 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:02.160649    2581 projected.go:196] Error preparing data for projected volume kube-api-access-2l7rn for pod openshift-network-diagnostics/network-check-target-z7848: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:02.160840 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:02.160713    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a99a3614-e2dc-4d83-9bed-153654b4bafe-kube-api-access-2l7rn podName:a99a3614-e2dc-4d83-9bed-153654b4bafe nodeName:}" failed. No retries permitted until 2026-09-17 17:57:10.160694292 +0000 UTC m=+18.251315926 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2l7rn" (UniqueName: "kubernetes.io/projected/a99a3614-e2dc-4d83-9bed-153654b4bafe-kube-api-access-2l7rn") pod "network-check-target-z7848" (UID: "a99a3614-e2dc-4d83-9bed-153654b4bafe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:02.354113 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:02.353266    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:57:02.354113 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:02.353693    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24pww" podUID="87bce58a-35d3-497c-8d82-7f925fe82b76"
Sep 17 17:57:02.354113 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:02.353857    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7848"
Sep 17 17:57:02.354113 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:02.354017    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7848" podUID="a99a3614-e2dc-4d83-9bed-153654b4bafe"
Sep 17 17:57:03.352283 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:03.352245    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:57:03.352467 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:03.352394    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xft5" podUID="33a2724c-596d-416e-99e2-583a9db84dcf"
Sep 17 17:57:04.352010 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:04.351976    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7848"
Sep 17 17:57:04.352010 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:04.352000    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:57:04.352476 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:04.352092    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7848" podUID="a99a3614-e2dc-4d83-9bed-153654b4bafe"
Sep 17 17:57:04.352476 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:04.352257    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24pww" podUID="87bce58a-35d3-497c-8d82-7f925fe82b76"
Sep 17 17:57:05.352815 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:05.352779    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:57:05.353283 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:05.352930    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xft5" podUID="33a2724c-596d-416e-99e2-583a9db84dcf"
Sep 17 17:57:06.352043 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:06.352003    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:57:06.352229 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:06.352003    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7848"
Sep 17 17:57:06.352229 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:06.352143    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24pww" podUID="87bce58a-35d3-497c-8d82-7f925fe82b76"
Sep 17 17:57:06.352321 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:06.352221    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7848" podUID="a99a3614-e2dc-4d83-9bed-153654b4bafe"
Sep 17 17:57:07.352129 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:07.352097    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:57:07.352590 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:07.352225    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xft5" podUID="33a2724c-596d-416e-99e2-583a9db84dcf"
Sep 17 17:57:08.352130 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:08.352093    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:57:08.352296 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:08.352094    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7848"
Sep 17 17:57:08.352296 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:08.352246    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24pww" podUID="87bce58a-35d3-497c-8d82-7f925fe82b76"
Sep 17 17:57:08.352296 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:08.352268    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7848" podUID="a99a3614-e2dc-4d83-9bed-153654b4bafe"
Sep 17 17:57:09.352631 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:09.352595    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:57:09.353180 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:09.352706    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xft5" podUID="33a2724c-596d-416e-99e2-583a9db84dcf"
Sep 17 17:57:10.119302 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:10.119264    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87bce58a-35d3-497c-8d82-7f925fe82b76-metrics-certs\") pod \"network-metrics-daemon-24pww\" (UID: \"87bce58a-35d3-497c-8d82-7f925fe82b76\") " pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:57:10.119503 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:10.119351    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/33a2724c-596d-416e-99e2-583a9db84dcf-original-pull-secret\") pod \"global-pull-secret-syncer-9xft5\" (UID: \"33a2724c-596d-416e-99e2-583a9db84dcf\") " pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:57:10.119503 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:10.119432    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:10.119503 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:10.119468    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:10.119630 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:10.119510    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87bce58a-35d3-497c-8d82-7f925fe82b76-metrics-certs podName:87bce58a-35d3-497c-8d82-7f925fe82b76 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:26.119487518 +0000 UTC m=+34.210109148 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87bce58a-35d3-497c-8d82-7f925fe82b76-metrics-certs") pod "network-metrics-daemon-24pww" (UID: "87bce58a-35d3-497c-8d82-7f925fe82b76") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:10.119630 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:10.119531    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33a2724c-596d-416e-99e2-583a9db84dcf-original-pull-secret podName:33a2724c-596d-416e-99e2-583a9db84dcf nodeName:}" failed. No retries permitted until 2026-09-17 17:57:26.119521812 +0000 UTC m=+34.210143440 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/33a2724c-596d-416e-99e2-583a9db84dcf-original-pull-secret") pod "global-pull-secret-syncer-9xft5" (UID: "33a2724c-596d-416e-99e2-583a9db84dcf") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:10.219859 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:10.219818    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l7rn\" (UniqueName: \"kubernetes.io/projected/a99a3614-e2dc-4d83-9bed-153654b4bafe-kube-api-access-2l7rn\") pod \"network-check-target-z7848\" (UID: \"a99a3614-e2dc-4d83-9bed-153654b4bafe\") " pod="openshift-network-diagnostics/network-check-target-z7848"
Sep 17 17:57:10.220030 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:10.219990    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:57:10.220030 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:10.220013    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:57:10.220030 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:10.220026    2581 projected.go:196] Error preparing data for projected volume kube-api-access-2l7rn for pod openshift-network-diagnostics/network-check-target-z7848: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:10.220174 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:10.220091    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a99a3614-e2dc-4d83-9bed-153654b4bafe-kube-api-access-2l7rn podName:a99a3614-e2dc-4d83-9bed-153654b4bafe nodeName:}" failed. No retries permitted until 2026-09-17 17:57:26.220072627 +0000 UTC m=+34.310694261 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2l7rn" (UniqueName: "kubernetes.io/projected/a99a3614-e2dc-4d83-9bed-153654b4bafe-kube-api-access-2l7rn") pod "network-check-target-z7848" (UID: "a99a3614-e2dc-4d83-9bed-153654b4bafe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:10.352464 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:10.352433    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:57:10.352630 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:10.352436    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7848"
Sep 17 17:57:10.352630 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:10.352544    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24pww" podUID="87bce58a-35d3-497c-8d82-7f925fe82b76"
Sep 17 17:57:10.353139 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:10.352631    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7848" podUID="a99a3614-e2dc-4d83-9bed-153654b4bafe"
Sep 17 17:57:11.352363 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:11.352325    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:57:11.352562 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:11.352452    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xft5" podUID="33a2724c-596d-416e-99e2-583a9db84dcf"
Sep 17 17:57:12.352741 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:12.352698    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7848"
Sep 17 17:57:12.353190 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:12.352808    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7848" podUID="a99a3614-e2dc-4d83-9bed-153654b4bafe"
Sep 17 17:57:12.353190 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:12.352824    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:57:12.353190 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:12.352984    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24pww" podUID="87bce58a-35d3-497c-8d82-7f925fe82b76"
Sep 17 17:57:13.354454 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:13.354197    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:57:13.354454 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:13.354217    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:57:13.354454 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:13.354317    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xft5" podUID="33a2724c-596d-416e-99e2-583a9db84dcf"
Sep 17 17:57:13.355070 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:13.354716    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24pww" podUID="87bce58a-35d3-497c-8d82-7f925fe82b76"
Sep 17 17:57:13.524025 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:13.523756    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2c98r" event={"ID":"f11a6327-77aa-444a-a4a4-abdddd8b9ec3","Type":"ContainerStarted","Data":"ed785221ac867f58a52873a9137e94dd6ec78b0c4757eb1e81811fe2dc56fd6e"}
Sep 17 17:57:13.526554 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:13.526518    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qcqn9" event={"ID":"e73ac43a-8a55-40ea-902c-5b06dfcc5d0b","Type":"ContainerStarted","Data":"d23bd429fa175c80b6748d435293ce803d03470cb5355d0738a434e1b9a6294a"}
Sep 17 17:57:13.528917 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:13.528875    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng" event={"ID":"041688ba-3c1a-426b-84b3-ec168a6a8658","Type":"ContainerStarted","Data":"d658882d26876bd48873d7f43488f6f7b30ed97921872f72b23178e8d89e4ab9"}
Sep 17 17:57:13.529055 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:13.528951    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng" event={"ID":"041688ba-3c1a-426b-84b3-ec168a6a8658","Type":"ContainerStarted","Data":"e02c82f9bb4d51dd94b4935cf98ff46245d7741f479a351bf4bd7cf1e832805b"}
Sep 17 17:57:13.530611 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:13.530580    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plwn4" event={"ID":"0f02a062-ac1a-4caf-a488-3543ce36210f","Type":"ContainerStarted","Data":"a8e75d70293184f2ab4a1b2e514a2d199f80b2263bbc71963d096a0071a27f3f"}
Sep 17 17:57:13.532229 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:13.532194    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zsz9h" event={"ID":"4383fe04-b7e0-4dd9-b3cc-ddca3d57aed2","Type":"ContainerStarted","Data":"ce3aaf80835ab057af62525642630cc04d4458c8f14c2fdc9d714937c1338246"}
Sep 17 17:57:13.533636 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:13.533616    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4nrxj" event={"ID":"e0c61526-ed8d-40e7-b87e-e9cb3ee45221","Type":"ContainerStarted","Data":"f1bbe79f527fa12c5d7d8fca9dea6a0f352e65b41bc8a95eae4b718c0ff9d7f1"}
Sep 17 17:57:13.535111 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:13.535089    2581 generic.go:356] "Generic (PLEG): container finished" podID="83940065-1b61-4aca-819e-102e270dffc4" containerID="19b91436c7d8c92f60ac567e310a21c98f785a8cc9d583eb1e59be6d97fd066a" exitCode=0
Sep 17 17:57:13.535242 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:13.535152    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v6f7b" event={"ID":"83940065-1b61-4aca-819e-102e270dffc4","Type":"ContainerDied","Data":"19b91436c7d8c92f60ac567e310a21c98f785a8cc9d583eb1e59be6d97fd066a"}
Sep 17 17:57:13.536616 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:13.536556    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8tpnw" event={"ID":"0f73bdd3-5b33-4d88-9d8b-eb540633a58e","Type":"ContainerStarted","Data":"af66b4a0be3d98011bf8aadef08482f7024bab26199f1fba5f97d64d9f3cc24a"}
Sep 17 17:57:13.547648 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:13.547603    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2c98r" podStartSLOduration=2.721890188 podStartE2EDuration="20.547586132s" totalImagesPullingTime="17.825695944s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:13.547359723 +0000 UTC m=+21.637981374" watchObservedRunningTime="2026-09-17 17:57:13.547586132 +0000 UTC m=+21.638207783"
Sep 17 17:57:13.548179 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:13.548144    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-28.ec2.internal" podStartSLOduration=19.548134961 podStartE2EDuration="19.548134961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:56:57.507234039 +0000 UTC m=+5.597855691" watchObservedRunningTime="2026-09-17 17:57:13.548134961 +0000 UTC m=+21.638756611"
Sep 17 17:57:13.600856 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:13.600812    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4nrxj" podStartSLOduration=3.65370302 podStartE2EDuration="21.600793804s" totalImagesPullingTime="17.947090784s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:13.572128212 +0000 UTC m=+21.662749862" watchObservedRunningTime="2026-09-17 17:57:13.600793804 +0000 UTC m=+21.691415455"
Sep 17 17:57:13.664987 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:13.664947    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8tpnw" podStartSLOduration=7.6252337820000005 podStartE2EDuration="21.664934002s" totalImagesPullingTime="14.03970022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:56:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:13.636541238 +0000 UTC m=+21.727162889" watchObservedRunningTime="2026-09-17 17:57:13.664934002 +0000 UTC m=+21.755555651"
Sep 17 17:57:13.687698 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:13.687661    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qcqn9" podStartSLOduration=7.610872129 podStartE2EDuration="21.687646893s" totalImagesPullingTime="14.076774764s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:13.667213824 +0000 UTC m=+21.757835476" watchObservedRunningTime="2026-09-17 17:57:13.687646893 +0000 UTC m=+21.778268543"
Sep 17 17:57:13.687786 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:13.687725    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zsz9h" podStartSLOduration=2.8972160540000003 podStartE2EDuration="20.68772023s" totalImagesPullingTime="17.790504176s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:13.687368317 +0000 UTC m=+21.777989966" watchObservedRunningTime="2026-09-17 17:57:13.68772023 +0000 UTC m=+21.778341897"
Sep 17 17:57:13.816423 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:13.816388    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8tpnw_0f73bdd3-5b33-4d88-9d8b-eb540633a58e/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:13.921552 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:13.921518    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8tpnw_0f73bdd3-5b33-4d88-9d8b-eb540633a58e/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:14.352582 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:14.352545    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7848"
Sep 17 17:57:14.352750 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:14.352681    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7848" podUID="a99a3614-e2dc-4d83-9bed-153654b4bafe"
Sep 17 17:57:14.545863 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:14.545825    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng" event={"ID":"041688ba-3c1a-426b-84b3-ec168a6a8658","Type":"ContainerStarted","Data":"513389a5d013f8dcd8397cd52e44cf83623c1527de6185898d07af949c77f808"}
Sep 17 17:57:14.546651 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:14.545928    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng" event={"ID":"041688ba-3c1a-426b-84b3-ec168a6a8658","Type":"ContainerStarted","Data":"83c92e5908e5dbf59ae80cf96a3d4ff61bdb4741bb7155585a0c7857d9c17eec"}
Sep 17 17:57:14.546651 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:14.545974    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng" event={"ID":"041688ba-3c1a-426b-84b3-ec168a6a8658","Type":"ContainerStarted","Data":"7fdaee4ef0720aed36085b4836df25c993d306beadf0344368fe616c33fa5181"}
Sep 17 17:57:14.546651 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:14.546012    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng" event={"ID":"041688ba-3c1a-426b-84b3-ec168a6a8658","Type":"ContainerStarted","Data":"89226a93f674e58d0f314003ad723cf9bb4aaac911f7b56b9b307779b67e2213"}
Sep 17 17:57:14.675126 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:14.674869    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 17:57:15.350525 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:15.350426    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T17:57:14.67510763Z","UUID":"b5ef7a1d-a9cb-4219-8d04-44defec3cf90","Handler":null,"Name":"","Endpoint":""}
Sep 17 17:57:15.351843 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:15.351818    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:57:15.352011 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:15.351829    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:57:15.352011 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:15.351947    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24pww" podUID="87bce58a-35d3-497c-8d82-7f925fe82b76"
Sep 17 17:57:15.352112 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:15.352013    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xft5" podUID="33a2724c-596d-416e-99e2-583a9db84dcf"
Sep 17 17:57:15.352196 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:15.352170    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 17:57:15.352245 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:15.352211    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 17:57:15.550160 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:15.550124    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z2vvf" event={"ID":"c1b1a6b8-52b7-45fb-9f31-5533cc2c85e8","Type":"ContainerStarted","Data":"bb284594ae06da4c2247aa009bc7113c8be31f406a30ecd0cdb00e9c31b755f9"}
Sep 17 17:57:15.552844 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:15.552811    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plwn4" event={"ID":"0f02a062-ac1a-4caf-a488-3543ce36210f","Type":"ContainerStarted","Data":"51995dffc09b477e963fa5e0ea4515a6f2261a503ca2856920ac56b46adef1e0"}
Sep 17 17:57:15.570493 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:15.570442    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-z2vvf" podStartSLOduration=5.721398355 podStartE2EDuration="23.570423169s" totalImagesPullingTime="17.849024814s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:15.570094731 +0000 UTC m=+23.660716382" watchObservedRunningTime="2026-09-17 17:57:15.570423169 +0000 UTC m=+23.661044820"
Sep 17 17:57:15.766191 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:15.766092    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8tpnw_0f73bdd3-5b33-4d88-9d8b-eb540633a58e/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:15.884145 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:15.884073    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8tpnw_0f73bdd3-5b33-4d88-9d8b-eb540633a58e/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:16.352756 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:16.352724    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7848"
Sep 17 17:57:16.353035 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:16.352839    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7848" podUID="a99a3614-e2dc-4d83-9bed-153654b4bafe"
Sep 17 17:57:16.495831 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:16.495798    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8tpnw_0f73bdd3-5b33-4d88-9d8b-eb540633a58e/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:16.559126 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:16.559094    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng" event={"ID":"041688ba-3c1a-426b-84b3-ec168a6a8658","Type":"ContainerStarted","Data":"a3e6e40687b755982cfef0bb04adf7483bd7cc2cd969b2d9012879bb9dff457a"}
Sep 17 17:57:16.561386 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:16.561363    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plwn4" event={"ID":"0f02a062-ac1a-4caf-a488-3543ce36210f","Type":"ContainerStarted","Data":"1d0b211235427d882ad7b30af04b6943b3c63277f17c87b378444c1226b5ce96"}
Sep 17 17:57:16.610446 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:16.610364    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8tpnw_0f73bdd3-5b33-4d88-9d8b-eb540633a58e/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:16.799534 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:16.799494    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8tpnw_0f73bdd3-5b33-4d88-9d8b-eb540633a58e/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:16.953616 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:16.953542    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8tpnw_0f73bdd3-5b33-4d88-9d8b-eb540633a58e/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:17.270483 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:17.270403    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8tpnw"
Sep 17 17:57:17.271074 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:17.271054    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8tpnw"
Sep 17 17:57:17.292548 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:17.292503    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-plwn4" podStartSLOduration=5.248138461 podStartE2EDuration="25.292487394s" totalImagesPullingTime="20.044348933s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:16.584070745 +0000 UTC m=+24.674692408" watchObservedRunningTime="2026-09-17 17:57:17.292487394 +0000 UTC m=+25.383109069"
Sep 17 17:57:17.352380 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:17.352346    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:57:17.352541 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:17.352491    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24pww" podUID="87bce58a-35d3-497c-8d82-7f925fe82b76"
Sep 17 17:57:17.352541 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:17.352519    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:57:17.352629 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:17.352604    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xft5" podUID="33a2724c-596d-416e-99e2-583a9db84dcf"
Sep 17 17:57:17.563745 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:17.563673    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8tpnw"
Sep 17 17:57:17.564207 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:17.564018    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8tpnw"
Sep 17 17:57:17.679319 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:17.679287    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8tpnw_0f73bdd3-5b33-4d88-9d8b-eb540633a58e/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:18.355768 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:18.355550    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7848"
Sep 17 17:57:18.355979 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:18.355797    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7848" podUID="a99a3614-e2dc-4d83-9bed-153654b4bafe"
Sep 17 17:57:18.569530 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:18.569493    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng" event={"ID":"041688ba-3c1a-426b-84b3-ec168a6a8658","Type":"ContainerStarted","Data":"ebabea4be2f4fee47fd336d897b690ad74afa96f2ad89e7e5157dc607947f928"}
Sep 17 17:57:18.570189 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:18.569743    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:57:18.570189 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:18.569809    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:57:18.571602 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:18.571577    2581 generic.go:356] "Generic (PLEG): container finished" podID="83940065-1b61-4aca-819e-102e270dffc4" containerID="aaf46fad79d5d99516b960614a310594d8ac6fe65c05c8263835936df71f779f" exitCode=0
Sep 17 17:57:18.571703 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:18.571658    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v6f7b" event={"ID":"83940065-1b61-4aca-819e-102e270dffc4","Type":"ContainerDied","Data":"aaf46fad79d5d99516b960614a310594d8ac6fe65c05c8263835936df71f779f"}
Sep 17 17:57:18.585167 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:18.585145    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:57:18.604582 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:18.604532    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng" podStartSLOduration=8.734409114 podStartE2EDuration="26.604517425s" totalImagesPullingTime="17.870108311s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:18.603933292 +0000 UTC m=+26.694554943" watchObservedRunningTime="2026-09-17 17:57:18.604517425 +0000 UTC m=+26.695139075"
Sep 17 17:57:19.352135 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:19.352036    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:57:19.352135 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:19.352087    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:57:19.352475 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:19.352193    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xft5" podUID="33a2724c-596d-416e-99e2-583a9db84dcf"
Sep 17 17:57:19.352475 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:19.352337    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24pww" podUID="87bce58a-35d3-497c-8d82-7f925fe82b76"
Sep 17 17:57:19.574828 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:19.574793    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:57:19.589994 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:19.589966    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:57:20.037566 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:20.037293    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9xft5"]
Sep 17 17:57:20.037725 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:20.037607    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:57:20.037794 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:20.037716    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xft5" podUID="33a2724c-596d-416e-99e2-583a9db84dcf"
Sep 17 17:57:20.039849 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:20.039821    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-24pww"]
Sep 17 17:57:20.039985 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:20.039951    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:57:20.040084 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:20.040060    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24pww" podUID="87bce58a-35d3-497c-8d82-7f925fe82b76"
Sep 17 17:57:20.040468 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:20.040451    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z7848"]
Sep 17 17:57:20.040535 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:20.040525    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7848"
Sep 17 17:57:20.040603 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:20.040587    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7848" podUID="a99a3614-e2dc-4d83-9bed-153654b4bafe"
Sep 17 17:57:20.578119 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:20.578087    2581 generic.go:356] "Generic (PLEG): container finished" podID="83940065-1b61-4aca-819e-102e270dffc4" containerID="f91a15535de04385c1de0a1b60a0786d7f3a904f346c32bb534606cc7bef799a" exitCode=0
Sep 17 17:57:20.578565 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:20.578173    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v6f7b" event={"ID":"83940065-1b61-4aca-819e-102e270dffc4","Type":"ContainerDied","Data":"f91a15535de04385c1de0a1b60a0786d7f3a904f346c32bb534606cc7bef799a"}
Sep 17 17:57:20.963578 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:20.963491    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8tpnw_0f73bdd3-5b33-4d88-9d8b-eb540633a58e/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:21.075003 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:21.074960    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8tpnw_0f73bdd3-5b33-4d88-9d8b-eb540633a58e/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:21.352418 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:21.352341    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:57:21.352553 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:21.352341    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7848"
Sep 17 17:57:21.352553 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:21.352458    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xft5" podUID="33a2724c-596d-416e-99e2-583a9db84dcf"
Sep 17 17:57:21.352553 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:21.352507    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7848" podUID="a99a3614-e2dc-4d83-9bed-153654b4bafe"
Sep 17 17:57:22.355666 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:22.354585    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:57:22.355666 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:22.354751    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24pww" podUID="87bce58a-35d3-497c-8d82-7f925fe82b76"
Sep 17 17:57:22.584254 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:22.584172    2581 generic.go:356] "Generic (PLEG): container finished" podID="83940065-1b61-4aca-819e-102e270dffc4" containerID="3bdfccde8819cf079343d9b99b2a85ca8fb8668a0ffcaf0090d39d3603c44a2c" exitCode=0
Sep 17 17:57:22.584254 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:22.584217    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v6f7b" event={"ID":"83940065-1b61-4aca-819e-102e270dffc4","Type":"ContainerDied","Data":"3bdfccde8819cf079343d9b99b2a85ca8fb8668a0ffcaf0090d39d3603c44a2c"}
Sep 17 17:57:23.351914 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:23.351867    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:57:23.352088 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:23.351926    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7848"
Sep 17 17:57:23.352088 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:23.352018    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xft5" podUID="33a2724c-596d-416e-99e2-583a9db84dcf"
Sep 17 17:57:23.352194 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:23.352160    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7848" podUID="a99a3614-e2dc-4d83-9bed-153654b4bafe"
Sep 17 17:57:24.352275 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:24.352244    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:57:24.352670 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:24.352399    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24pww" podUID="87bce58a-35d3-497c-8d82-7f925fe82b76"
Sep 17 17:57:25.352594 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:25.352567    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:57:25.353106 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:25.353073    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9xft5" podUID="33a2724c-596d-416e-99e2-583a9db84dcf"
Sep 17 17:57:25.353182 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:25.353133    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7848"
Sep 17 17:57:25.353337 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:25.353310    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7848" podUID="a99a3614-e2dc-4d83-9bed-153654b4bafe"
Sep 17 17:57:26.140880 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:26.140836    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/33a2724c-596d-416e-99e2-583a9db84dcf-original-pull-secret\") pod \"global-pull-secret-syncer-9xft5\" (UID: \"33a2724c-596d-416e-99e2-583a9db84dcf\") " pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:57:26.141075 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:26.140914    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87bce58a-35d3-497c-8d82-7f925fe82b76-metrics-certs\") pod \"network-metrics-daemon-24pww\" (UID: \"87bce58a-35d3-497c-8d82-7f925fe82b76\") " pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:57:26.141075 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:26.141019    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:26.141075 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:26.141055    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:26.141217 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:26.141104    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33a2724c-596d-416e-99e2-583a9db84dcf-original-pull-secret podName:33a2724c-596d-416e-99e2-583a9db84dcf nodeName:}" failed. No retries permitted until 2026-09-17 17:57:58.141082197 +0000 UTC m=+66.231703840 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/33a2724c-596d-416e-99e2-583a9db84dcf-original-pull-secret") pod "global-pull-secret-syncer-9xft5" (UID: "33a2724c-596d-416e-99e2-583a9db84dcf") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:57:26.141217 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:26.141125    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87bce58a-35d3-497c-8d82-7f925fe82b76-metrics-certs podName:87bce58a-35d3-497c-8d82-7f925fe82b76 nodeName:}" failed. No retries permitted until 2026-09-17 17:57:58.141114528 +0000 UTC m=+66.231736160 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87bce58a-35d3-497c-8d82-7f925fe82b76-metrics-certs") pod "network-metrics-daemon-24pww" (UID: "87bce58a-35d3-497c-8d82-7f925fe82b76") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:57:26.241832 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:26.241791    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l7rn\" (UniqueName: \"kubernetes.io/projected/a99a3614-e2dc-4d83-9bed-153654b4bafe-kube-api-access-2l7rn\") pod \"network-check-target-z7848\" (UID: \"a99a3614-e2dc-4d83-9bed-153654b4bafe\") " pod="openshift-network-diagnostics/network-check-target-z7848"
Sep 17 17:57:26.242021 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:26.241966    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:57:26.242021 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:26.241990    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:57:26.242021 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:26.242004    2581 projected.go:196] Error preparing data for projected volume kube-api-access-2l7rn for pod openshift-network-diagnostics/network-check-target-z7848: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:26.242159 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:26.242086    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a99a3614-e2dc-4d83-9bed-153654b4bafe-kube-api-access-2l7rn podName:a99a3614-e2dc-4d83-9bed-153654b4bafe nodeName:}" failed. No retries permitted until 2026-09-17 17:57:58.242065938 +0000 UTC m=+66.332687567 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2l7rn" (UniqueName: "kubernetes.io/projected/a99a3614-e2dc-4d83-9bed-153654b4bafe-kube-api-access-2l7rn") pod "network-check-target-z7848" (UID: "a99a3614-e2dc-4d83-9bed-153654b4bafe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:57:26.352485 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:26.352448    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:57:26.352651 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:26.352599    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-24pww" podUID="87bce58a-35d3-497c-8d82-7f925fe82b76"
Sep 17 17:57:26.597470 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:26.597437    2581 generic.go:356] "Generic (PLEG): container finished" podID="83940065-1b61-4aca-819e-102e270dffc4" containerID="f6fb7098079efb7f085b1551a6fe42215461fb5aaef4a4bc6c2eeeed8f7c87d8" exitCode=0
Sep 17 17:57:26.597617 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:26.597498    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v6f7b" event={"ID":"83940065-1b61-4aca-819e-102e270dffc4","Type":"ContainerDied","Data":"f6fb7098079efb7f085b1551a6fe42215461fb5aaef4a4bc6c2eeeed8f7c87d8"}
Sep 17 17:57:27.210836 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.210807    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-28.ec2.internal" event="NodeReady"
Sep 17 17:57:27.211014 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.210958    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 17:57:27.276687 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.276653    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sgwxz"]
Sep 17 17:57:27.278705 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.278691    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sgwxz"
Sep 17 17:57:27.281985 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.281962    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 17:57:27.282179 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.282161    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 17:57:27.282270 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.282253    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4ffrq\""
Sep 17 17:57:27.283446 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.283426    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6nv7n"]
Sep 17 17:57:27.285481 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.285361    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6nv7n"
Sep 17 17:57:27.289568 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.289545    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 17:57:27.289840 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.289823    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 17:57:27.290089 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.290074    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mkswj\""
Sep 17 17:57:27.290393 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.290376    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 17:57:27.290863 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.290846    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sgwxz"]
Sep 17 17:57:27.305403 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.305374    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6nv7n"]
Sep 17 17:57:27.351833 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.351805    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:57:27.352005 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.351949    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7848"
Sep 17 17:57:27.355252 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.355209    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-m4z79\""
Sep 17 17:57:27.355252 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.355244    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:57:27.355646 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.355382    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:57:27.355646 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.355491    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:57:27.451599 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.451563    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7whm\" (UniqueName: \"kubernetes.io/projected/06dbe6ba-8122-4c23-9dbe-63e2de449f7c-kube-api-access-p7whm\") pod \"ingress-canary-6nv7n\" (UID: \"06dbe6ba-8122-4c23-9dbe-63e2de449f7c\") " pod="openshift-ingress-canary/ingress-canary-6nv7n"
Sep 17 17:57:27.451771 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.451616    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c59508-5d4e-4ea8-a211-42ec8c5bfffa-tmp-dir\") pod \"dns-default-sgwxz\" (UID: \"13c59508-5d4e-4ea8-a211-42ec8c5bfffa\") " pod="openshift-dns/dns-default-sgwxz"
Sep 17 17:57:27.451771 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.451649    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13c59508-5d4e-4ea8-a211-42ec8c5bfffa-metrics-tls\") pod \"dns-default-sgwxz\" (UID: \"13c59508-5d4e-4ea8-a211-42ec8c5bfffa\") " pod="openshift-dns/dns-default-sgwxz"
Sep 17 17:57:27.451771 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.451705    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/06dbe6ba-8122-4c23-9dbe-63e2de449f7c-cert\") pod \"ingress-canary-6nv7n\" (UID: \"06dbe6ba-8122-4c23-9dbe-63e2de449f7c\") " pod="openshift-ingress-canary/ingress-canary-6nv7n"
Sep 17 17:57:27.451771 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.451749    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/13c59508-5d4e-4ea8-a211-42ec8c5bfffa-config-volume\") pod \"dns-default-sgwxz\" (UID: \"13c59508-5d4e-4ea8-a211-42ec8c5bfffa\") " pod="openshift-dns/dns-default-sgwxz"
Sep 17 17:57:27.451892 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.451776    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94v5w\" (UniqueName: \"kubernetes.io/projected/13c59508-5d4e-4ea8-a211-42ec8c5bfffa-kube-api-access-94v5w\") pod \"dns-default-sgwxz\" (UID: \"13c59508-5d4e-4ea8-a211-42ec8c5bfffa\") " pod="openshift-dns/dns-default-sgwxz"
Sep 17 17:57:27.516290 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.516206    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8tpnw_0f73bdd3-5b33-4d88-9d8b-eb540633a58e/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:27.552274 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.552247    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7whm\" (UniqueName: \"kubernetes.io/projected/06dbe6ba-8122-4c23-9dbe-63e2de449f7c-kube-api-access-p7whm\") pod \"ingress-canary-6nv7n\" (UID: \"06dbe6ba-8122-4c23-9dbe-63e2de449f7c\") " pod="openshift-ingress-canary/ingress-canary-6nv7n"
Sep 17 17:57:27.552396 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.552294    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c59508-5d4e-4ea8-a211-42ec8c5bfffa-tmp-dir\") pod \"dns-default-sgwxz\" (UID: \"13c59508-5d4e-4ea8-a211-42ec8c5bfffa\") " pod="openshift-dns/dns-default-sgwxz"
Sep 17 17:57:27.552396 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.552327    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13c59508-5d4e-4ea8-a211-42ec8c5bfffa-metrics-tls\") pod \"dns-default-sgwxz\" (UID: \"13c59508-5d4e-4ea8-a211-42ec8c5bfffa\") " pod="openshift-dns/dns-default-sgwxz"
Sep 17 17:57:27.552470 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:27.552429    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:27.552470 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.552452    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/06dbe6ba-8122-4c23-9dbe-63e2de449f7c-cert\") pod \"ingress-canary-6nv7n\" (UID: \"06dbe6ba-8122-4c23-9dbe-63e2de449f7c\") " pod="openshift-ingress-canary/ingress-canary-6nv7n"
Sep 17 17:57:27.552545 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:27.552489    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13c59508-5d4e-4ea8-a211-42ec8c5bfffa-metrics-tls podName:13c59508-5d4e-4ea8-a211-42ec8c5bfffa nodeName:}" failed. No retries permitted until 2026-09-17 17:57:28.052472518 +0000 UTC m=+36.143094148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/13c59508-5d4e-4ea8-a211-42ec8c5bfffa-metrics-tls") pod "dns-default-sgwxz" (UID: "13c59508-5d4e-4ea8-a211-42ec8c5bfffa") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:27.552545 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.552525    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/13c59508-5d4e-4ea8-a211-42ec8c5bfffa-config-volume\") pod \"dns-default-sgwxz\" (UID: \"13c59508-5d4e-4ea8-a211-42ec8c5bfffa\") " pod="openshift-dns/dns-default-sgwxz"
Sep 17 17:57:27.552626 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.552551    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94v5w\" (UniqueName: \"kubernetes.io/projected/13c59508-5d4e-4ea8-a211-42ec8c5bfffa-kube-api-access-94v5w\") pod \"dns-default-sgwxz\" (UID: \"13c59508-5d4e-4ea8-a211-42ec8c5bfffa\") " pod="openshift-dns/dns-default-sgwxz"
Sep 17 17:57:27.552681 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.552661    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13c59508-5d4e-4ea8-a211-42ec8c5bfffa-tmp-dir\") pod \"dns-default-sgwxz\" (UID: \"13c59508-5d4e-4ea8-a211-42ec8c5bfffa\") " pod="openshift-dns/dns-default-sgwxz"
Sep 17 17:57:27.553197 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.553178    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/13c59508-5d4e-4ea8-a211-42ec8c5bfffa-config-volume\") pod \"dns-default-sgwxz\" (UID: \"13c59508-5d4e-4ea8-a211-42ec8c5bfffa\") " pod="openshift-dns/dns-default-sgwxz"
Sep 17 17:57:27.556775 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.556753    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/06dbe6ba-8122-4c23-9dbe-63e2de449f7c-cert\") pod \"ingress-canary-6nv7n\" (UID: \"06dbe6ba-8122-4c23-9dbe-63e2de449f7c\") " pod="openshift-ingress-canary/ingress-canary-6nv7n"
Sep 17 17:57:27.563791 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.563764    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94v5w\" (UniqueName: \"kubernetes.io/projected/13c59508-5d4e-4ea8-a211-42ec8c5bfffa-kube-api-access-94v5w\") pod \"dns-default-sgwxz\" (UID: \"13c59508-5d4e-4ea8-a211-42ec8c5bfffa\") " pod="openshift-dns/dns-default-sgwxz"
Sep 17 17:57:27.563887 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.563837    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7whm\" (UniqueName: \"kubernetes.io/projected/06dbe6ba-8122-4c23-9dbe-63e2de449f7c-kube-api-access-p7whm\") pod \"ingress-canary-6nv7n\" (UID: \"06dbe6ba-8122-4c23-9dbe-63e2de449f7c\") " pod="openshift-ingress-canary/ingress-canary-6nv7n"
Sep 17 17:57:27.598004 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.597966    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6nv7n"
Sep 17 17:57:27.603111 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.603085    2581 generic.go:356] "Generic (PLEG): container finished" podID="83940065-1b61-4aca-819e-102e270dffc4" containerID="b6091d4fe676d62426f6797dc98b02bab9e4741bf1a1036a3d8b76ab9a4b1186" exitCode=0
Sep 17 17:57:27.603217 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.603121    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v6f7b" event={"ID":"83940065-1b61-4aca-819e-102e270dffc4","Type":"ContainerDied","Data":"b6091d4fe676d62426f6797dc98b02bab9e4741bf1a1036a3d8b76ab9a4b1186"}
Sep 17 17:57:27.890452 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:27.890279    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6nv7n"]
Sep 17 17:57:27.894088 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:57:27.894037    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06dbe6ba_8122_4c23_9dbe_63e2de449f7c.slice/crio-ccf4cf23bd112a39a713886eff3914e5d92250a617ca1735525a4d1f885ed78a WatchSource:0}: Error finding container ccf4cf23bd112a39a713886eff3914e5d92250a617ca1735525a4d1f885ed78a: Status 404 returned error can't find the container with id ccf4cf23bd112a39a713886eff3914e5d92250a617ca1735525a4d1f885ed78a
Sep 17 17:57:28.056689 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:28.056657    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13c59508-5d4e-4ea8-a211-42ec8c5bfffa-metrics-tls\") pod \"dns-default-sgwxz\" (UID: \"13c59508-5d4e-4ea8-a211-42ec8c5bfffa\") " pod="openshift-dns/dns-default-sgwxz"
Sep 17 17:57:28.056866 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:28.056823    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:28.056932 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:28.056887    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13c59508-5d4e-4ea8-a211-42ec8c5bfffa-metrics-tls podName:13c59508-5d4e-4ea8-a211-42ec8c5bfffa nodeName:}" failed. No retries permitted until 2026-09-17 17:57:29.056870982 +0000 UTC m=+37.147492627 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/13c59508-5d4e-4ea8-a211-42ec8c5bfffa-metrics-tls") pod "dns-default-sgwxz" (UID: "13c59508-5d4e-4ea8-a211-42ec8c5bfffa") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:28.351989 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:28.351895    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:57:28.354701 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:28.354673    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cf4sb\""
Sep 17 17:57:28.354701 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:28.354679    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:57:28.610180 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:28.610096    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v6f7b" event={"ID":"83940065-1b61-4aca-819e-102e270dffc4","Type":"ContainerStarted","Data":"60367d7248db8c31f4ffb463d00e1022d78113f507de92abddfa457555c03465"}
Sep 17 17:57:28.611469 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:28.611444    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6nv7n" event={"ID":"06dbe6ba-8122-4c23-9dbe-63e2de449f7c","Type":"ContainerStarted","Data":"ccf4cf23bd112a39a713886eff3914e5d92250a617ca1735525a4d1f885ed78a"}
Sep 17 17:57:28.639646 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:28.639584    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-v6f7b" podStartSLOduration=8.652039193 podStartE2EDuration="36.639565211s" totalImagesPullingTime="27.987526018s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:28.639140211 +0000 UTC m=+36.729761862" watchObservedRunningTime="2026-09-17 17:57:28.639565211 +0000 UTC m=+36.730186862"
Sep 17 17:57:29.064573 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:29.064488    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13c59508-5d4e-4ea8-a211-42ec8c5bfffa-metrics-tls\") pod \"dns-default-sgwxz\" (UID: \"13c59508-5d4e-4ea8-a211-42ec8c5bfffa\") " pod="openshift-dns/dns-default-sgwxz"
Sep 17 17:57:29.064732 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:29.064680    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:29.064783 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:29.064747    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13c59508-5d4e-4ea8-a211-42ec8c5bfffa-metrics-tls podName:13c59508-5d4e-4ea8-a211-42ec8c5bfffa nodeName:}" failed. No retries permitted until 2026-09-17 17:57:31.064731614 +0000 UTC m=+39.155353247 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/13c59508-5d4e-4ea8-a211-42ec8c5bfffa-metrics-tls") pod "dns-default-sgwxz" (UID: "13c59508-5d4e-4ea8-a211-42ec8c5bfffa") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:30.618463 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:30.618426    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6nv7n" event={"ID":"06dbe6ba-8122-4c23-9dbe-63e2de449f7c","Type":"ContainerStarted","Data":"e083ecaf523552501240b6e2830b857fea758881472c94e57d1ce99d41eacabd"}
Sep 17 17:57:30.640685 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:30.640564    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6nv7n" podStartSLOduration=1.173420327 podStartE2EDuration="3.640547755s" totalImagesPullingTime="2.467127428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:30.64031587 +0000 UTC m=+38.730937521" watchObservedRunningTime="2026-09-17 17:57:30.640547755 +0000 UTC m=+38.731169402"
Sep 17 17:57:31.078131 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:31.078038    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13c59508-5d4e-4ea8-a211-42ec8c5bfffa-metrics-tls\") pod \"dns-default-sgwxz\" (UID: \"13c59508-5d4e-4ea8-a211-42ec8c5bfffa\") " pod="openshift-dns/dns-default-sgwxz"
Sep 17 17:57:31.078283 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:31.078166    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:31.078283 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:31.078217    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13c59508-5d4e-4ea8-a211-42ec8c5bfffa-metrics-tls podName:13c59508-5d4e-4ea8-a211-42ec8c5bfffa nodeName:}" failed. No retries permitted until 2026-09-17 17:57:35.078204764 +0000 UTC m=+43.168826392 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/13c59508-5d4e-4ea8-a211-42ec8c5bfffa-metrics-tls") pod "dns-default-sgwxz" (UID: "13c59508-5d4e-4ea8-a211-42ec8c5bfffa") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:35.107866 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:35.107825    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13c59508-5d4e-4ea8-a211-42ec8c5bfffa-metrics-tls\") pod \"dns-default-sgwxz\" (UID: \"13c59508-5d4e-4ea8-a211-42ec8c5bfffa\") " pod="openshift-dns/dns-default-sgwxz"
Sep 17 17:57:35.108279 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:35.107957    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:35.108279 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:35.108010    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13c59508-5d4e-4ea8-a211-42ec8c5bfffa-metrics-tls podName:13c59508-5d4e-4ea8-a211-42ec8c5bfffa nodeName:}" failed. No retries permitted until 2026-09-17 17:57:43.107996396 +0000 UTC m=+51.198618024 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/13c59508-5d4e-4ea8-a211-42ec8c5bfffa-metrics-tls") pod "dns-default-sgwxz" (UID: "13c59508-5d4e-4ea8-a211-42ec8c5bfffa") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:43.162688 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:43.162652    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13c59508-5d4e-4ea8-a211-42ec8c5bfffa-metrics-tls\") pod \"dns-default-sgwxz\" (UID: \"13c59508-5d4e-4ea8-a211-42ec8c5bfffa\") " pod="openshift-dns/dns-default-sgwxz"
Sep 17 17:57:43.163140 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:43.162803    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:43.163140 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:43.162882    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13c59508-5d4e-4ea8-a211-42ec8c5bfffa-metrics-tls podName:13c59508-5d4e-4ea8-a211-42ec8c5bfffa nodeName:}" failed. No retries permitted until 2026-09-17 17:57:59.162862785 +0000 UTC m=+67.253484423 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/13c59508-5d4e-4ea8-a211-42ec8c5bfffa-metrics-tls") pod "dns-default-sgwxz" (UID: "13c59508-5d4e-4ea8-a211-42ec8c5bfffa") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:51.596874 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:51.596845    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bz9ng"
Sep 17 17:57:58.169587 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:58.169542    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/33a2724c-596d-416e-99e2-583a9db84dcf-original-pull-secret\") pod \"global-pull-secret-syncer-9xft5\" (UID: \"33a2724c-596d-416e-99e2-583a9db84dcf\") " pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:57:58.169587 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:58.169596    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87bce58a-35d3-497c-8d82-7f925fe82b76-metrics-certs\") pod \"network-metrics-daemon-24pww\" (UID: \"87bce58a-35d3-497c-8d82-7f925fe82b76\") " pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:57:58.177178 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:58.177156    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:57:58.177256 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:58.177160    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:57:58.180250 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:58.180233    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 17:57:58.180345 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:58.180301    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87bce58a-35d3-497c-8d82-7f925fe82b76-metrics-certs podName:87bce58a-35d3-497c-8d82-7f925fe82b76 nodeName:}" failed. No retries permitted until 2026-09-17 17:59:02.180280978 +0000 UTC m=+130.270902624 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87bce58a-35d3-497c-8d82-7f925fe82b76-metrics-certs") pod "network-metrics-daemon-24pww" (UID: "87bce58a-35d3-497c-8d82-7f925fe82b76") : secret "metrics-daemon-secret" not found
Sep 17 17:57:58.183341 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:58.183309    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/33a2724c-596d-416e-99e2-583a9db84dcf-original-pull-secret\") pod \"global-pull-secret-syncer-9xft5\" (UID: \"33a2724c-596d-416e-99e2-583a9db84dcf\") " pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:57:58.261797 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:58.261762    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9xft5"
Sep 17 17:57:58.270349 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:58.270324    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l7rn\" (UniqueName: \"kubernetes.io/projected/a99a3614-e2dc-4d83-9bed-153654b4bafe-kube-api-access-2l7rn\") pod \"network-check-target-z7848\" (UID: \"a99a3614-e2dc-4d83-9bed-153654b4bafe\") " pod="openshift-network-diagnostics/network-check-target-z7848"
Sep 17 17:57:58.277417 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:58.277395    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:57:58.284076 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:58.284057    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:57:58.294195 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:58.294176    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l7rn\" (UniqueName: \"kubernetes.io/projected/a99a3614-e2dc-4d83-9bed-153654b4bafe-kube-api-access-2l7rn\") pod \"network-check-target-z7848\" (UID: \"a99a3614-e2dc-4d83-9bed-153654b4bafe\") " pod="openshift-network-diagnostics/network-check-target-z7848"
Sep 17 17:57:58.558597 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:58.558559    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9xft5"]
Sep 17 17:57:58.561555 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:57:58.561527    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a2724c_596d_416e_99e2_583a9db84dcf.slice/crio-181a11b363ba8fc2a40fb5c1fe50f4f06f2a14fdd651f3d3376c2a71c43a2890 WatchSource:0}: Error finding container 181a11b363ba8fc2a40fb5c1fe50f4f06f2a14fdd651f3d3376c2a71c43a2890: Status 404 returned error can't find the container with id 181a11b363ba8fc2a40fb5c1fe50f4f06f2a14fdd651f3d3376c2a71c43a2890
Sep 17 17:57:58.568530 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:58.568509    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-m4z79\""
Sep 17 17:57:58.576001 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:58.575959    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7848"
Sep 17 17:57:58.687959 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:58.687929    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9xft5" event={"ID":"33a2724c-596d-416e-99e2-583a9db84dcf","Type":"ContainerStarted","Data":"181a11b363ba8fc2a40fb5c1fe50f4f06f2a14fdd651f3d3376c2a71c43a2890"}
Sep 17 17:57:58.879926 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:58.879886    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z7848"]
Sep 17 17:57:58.883413 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:57:58.883388    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda99a3614_e2dc_4d83_9bed_153654b4bafe.slice/crio-eb261100f7b720e62bb15b1a8cf0aef4a6b22cae3609cd7e568fa861e0b42716 WatchSource:0}: Error finding container eb261100f7b720e62bb15b1a8cf0aef4a6b22cae3609cd7e568fa861e0b42716: Status 404 returned error can't find the container with id eb261100f7b720e62bb15b1a8cf0aef4a6b22cae3609cd7e568fa861e0b42716
Sep 17 17:57:59.176241 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:59.176149    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13c59508-5d4e-4ea8-a211-42ec8c5bfffa-metrics-tls\") pod \"dns-default-sgwxz\" (UID: \"13c59508-5d4e-4ea8-a211-42ec8c5bfffa\") " pod="openshift-dns/dns-default-sgwxz"
Sep 17 17:57:59.176625 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:59.176317    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:57:59.176625 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:57:59.176391    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13c59508-5d4e-4ea8-a211-42ec8c5bfffa-metrics-tls podName:13c59508-5d4e-4ea8-a211-42ec8c5bfffa nodeName:}" failed. No retries permitted until 2026-09-17 17:58:31.176370228 +0000 UTC m=+99.266991855 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/13c59508-5d4e-4ea8-a211-42ec8c5bfffa-metrics-tls") pod "dns-default-sgwxz" (UID: "13c59508-5d4e-4ea8-a211-42ec8c5bfffa") : secret "dns-default-metrics-tls" not found
Sep 17 17:57:59.692506 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:57:59.692431    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z7848" event={"ID":"a99a3614-e2dc-4d83-9bed-153654b4bafe","Type":"ContainerStarted","Data":"eb261100f7b720e62bb15b1a8cf0aef4a6b22cae3609cd7e568fa861e0b42716"}
Sep 17 17:58:00.705180 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.705145    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zt5hc"]
Sep 17 17:58:00.707075 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.707049    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zt5hc"
Sep 17 17:58:00.717553 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.717518    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 17:58:00.717773 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.717742    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-lm8ds\""
Sep 17 17:58:00.717863 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.717846    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 17 17:58:00.718162 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.718144    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 17:58:00.718896 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.718716    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 17 17:58:00.719040 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.719012    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-lkct8"]
Sep 17 17:58:00.722375 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.722338    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-lkct8"
Sep 17 17:58:00.729232 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.729209    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-sbsm6\""
Sep 17 17:58:00.729705 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.729534    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 17 17:58:00.729705 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.729538    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 17 17:58:00.730225 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.730208    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 17 17:58:00.730441 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.730426    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:58:00.739048 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.739023    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zt5hc"]
Sep 17 17:58:00.740329 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.740274    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-lkct8"]
Sep 17 17:58:00.788563 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.788524    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9a99ade3-9dfb-4718-bf9b-07d713715bab-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-zt5hc\" (UID: \"9a99ade3-9dfb-4718-bf9b-07d713715bab\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zt5hc"
Sep 17 17:58:00.788563 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.788572    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a99ade3-9dfb-4718-bf9b-07d713715bab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-zt5hc\" (UID: \"9a99ade3-9dfb-4718-bf9b-07d713715bab\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zt5hc"
Sep 17 17:58:00.788825 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.788599    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/08540561-3f4b-46ef-b859-d132f8488e4c-config\") pod \"service-ca-operator-57cdc4b768-lkct8\" (UID: \"08540561-3f4b-46ef-b859-d132f8488e4c\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-lkct8"
Sep 17 17:58:00.788825 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.788658    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-496br\" (UniqueName: \"kubernetes.io/projected/9a99ade3-9dfb-4718-bf9b-07d713715bab-kube-api-access-496br\") pod \"cluster-monitoring-operator-847b8bd8d7-zt5hc\" (UID: \"9a99ade3-9dfb-4718-bf9b-07d713715bab\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zt5hc"
Sep 17 17:58:00.788825 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.788706    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/08540561-3f4b-46ef-b859-d132f8488e4c-serving-cert\") pod \"service-ca-operator-57cdc4b768-lkct8\" (UID: \"08540561-3f4b-46ef-b859-d132f8488e4c\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-lkct8"
Sep 17 17:58:00.788825 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.788753    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r2lv\" (UniqueName: \"kubernetes.io/projected/08540561-3f4b-46ef-b859-d132f8488e4c-kube-api-access-9r2lv\") pod \"service-ca-operator-57cdc4b768-lkct8\" (UID: \"08540561-3f4b-46ef-b859-d132f8488e4c\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-lkct8"
Sep 17 17:58:00.814958 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.814925    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gw2lt"]
Sep 17 17:58:00.818974 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.817985    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gw2lt"
Sep 17 17:58:00.819238 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.819212    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-l5vq7"]
Sep 17 17:58:00.821139 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.821117    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-l5vq7"
Sep 17 17:58:00.822432 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.822363    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 17 17:58:00.822432 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.822416    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-bld98\""
Sep 17 17:58:00.822630 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.822622    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 17 17:58:00.822858 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.822840    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:58:00.824621 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.824603    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-4jlkv\""
Sep 17 17:58:00.824715 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.824654    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:58:00.825398 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.825377    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 17 17:58:00.825484 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.825410    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 17 17:58:00.826150 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.826129    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 17 17:58:00.833450 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.833426    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 17 17:58:00.848379 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.848307    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gw2lt"]
Sep 17 17:58:00.862716 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.862682    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-l5vq7"]
Sep 17 17:58:00.889296 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.889204    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b41e1ab1-9be9-4db0-82da-f3dc6d863a24-serving-cert\") pod \"console-operator-fd4bb46c-l5vq7\" (UID: \"b41e1ab1-9be9-4db0-82da-f3dc6d863a24\") " pod="openshift-console-operator/console-operator-fd4bb46c-l5vq7"
Sep 17 17:58:00.889296 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.889275    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9a99ade3-9dfb-4718-bf9b-07d713715bab-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-zt5hc\" (UID: \"9a99ade3-9dfb-4718-bf9b-07d713715bab\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zt5hc"
Sep 17 17:58:00.889568 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.889308    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a99ade3-9dfb-4718-bf9b-07d713715bab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-zt5hc\" (UID: \"9a99ade3-9dfb-4718-bf9b-07d713715bab\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zt5hc"
Sep 17 17:58:00.889568 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.889366    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/08540561-3f4b-46ef-b859-d132f8488e4c-config\") pod \"service-ca-operator-57cdc4b768-lkct8\" (UID: \"08540561-3f4b-46ef-b859-d132f8488e4c\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-lkct8"
Sep 17 17:58:00.889568 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.889413    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b41e1ab1-9be9-4db0-82da-f3dc6d863a24-trusted-ca\") pod \"console-operator-fd4bb46c-l5vq7\" (UID: \"b41e1ab1-9be9-4db0-82da-f3dc6d863a24\") " pod="openshift-console-operator/console-operator-fd4bb46c-l5vq7"
Sep 17 17:58:00.889568 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.889446    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmsvx\" (UniqueName: \"kubernetes.io/projected/c8209260-6dd4-4d36-a654-d4675c37356b-kube-api-access-wmsvx\") pod \"cluster-samples-operator-685d7d68b5-gw2lt\" (UID: \"c8209260-6dd4-4d36-a654-d4675c37356b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gw2lt"
Sep 17 17:58:00.889568 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.889480    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-496br\" (UniqueName: \"kubernetes.io/projected/9a99ade3-9dfb-4718-bf9b-07d713715bab-kube-api-access-496br\") pod \"cluster-monitoring-operator-847b8bd8d7-zt5hc\" (UID: \"9a99ade3-9dfb-4718-bf9b-07d713715bab\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zt5hc"
Sep 17 17:58:00.889568 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.889539    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/08540561-3f4b-46ef-b859-d132f8488e4c-serving-cert\") pod \"service-ca-operator-57cdc4b768-lkct8\" (UID: \"08540561-3f4b-46ef-b859-d132f8488e4c\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-lkct8"
Sep 17 17:58:00.889858 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.889598    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b41e1ab1-9be9-4db0-82da-f3dc6d863a24-config\") pod \"console-operator-fd4bb46c-l5vq7\" (UID: \"b41e1ab1-9be9-4db0-82da-f3dc6d863a24\") " pod="openshift-console-operator/console-operator-fd4bb46c-l5vq7"
Sep 17 17:58:00.889858 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.889646    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r2lv\" (UniqueName: \"kubernetes.io/projected/08540561-3f4b-46ef-b859-d132f8488e4c-kube-api-access-9r2lv\") pod \"service-ca-operator-57cdc4b768-lkct8\" (UID: \"08540561-3f4b-46ef-b859-d132f8488e4c\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-lkct8"
Sep 17 17:58:00.889858 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.889681    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rwlr\" (UniqueName: \"kubernetes.io/projected/b41e1ab1-9be9-4db0-82da-f3dc6d863a24-kube-api-access-9rwlr\") pod \"console-operator-fd4bb46c-l5vq7\" (UID: \"b41e1ab1-9be9-4db0-82da-f3dc6d863a24\") " pod="openshift-console-operator/console-operator-fd4bb46c-l5vq7"
Sep 17 17:58:00.889858 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.889706    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c8209260-6dd4-4d36-a654-d4675c37356b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-gw2lt\" (UID: \"c8209260-6dd4-4d36-a654-d4675c37356b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gw2lt"
Sep 17 17:58:00.890091 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:00.889871    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:58:00.890091 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:00.889956    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a99ade3-9dfb-4718-bf9b-07d713715bab-cluster-monitoring-operator-tls podName:9a99ade3-9dfb-4718-bf9b-07d713715bab nodeName:}" failed. No retries permitted until 2026-09-17 17:58:01.389934479 +0000 UTC m=+69.480556119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9a99ade3-9dfb-4718-bf9b-07d713715bab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-zt5hc" (UID: "9a99ade3-9dfb-4718-bf9b-07d713715bab") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:58:00.890207 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.890167    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9a99ade3-9dfb-4718-bf9b-07d713715bab-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-zt5hc\" (UID: \"9a99ade3-9dfb-4718-bf9b-07d713715bab\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zt5hc"
Sep 17 17:58:00.890262 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.890242    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/08540561-3f4b-46ef-b859-d132f8488e4c-config\") pod \"service-ca-operator-57cdc4b768-lkct8\" (UID: \"08540561-3f4b-46ef-b859-d132f8488e4c\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-lkct8"
Sep 17 17:58:00.892568 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.892538    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/08540561-3f4b-46ef-b859-d132f8488e4c-serving-cert\") pod \"service-ca-operator-57cdc4b768-lkct8\" (UID: \"08540561-3f4b-46ef-b859-d132f8488e4c\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-lkct8"
Sep 17 17:58:00.899468 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.899444    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r2lv\" (UniqueName: \"kubernetes.io/projected/08540561-3f4b-46ef-b859-d132f8488e4c-kube-api-access-9r2lv\") pod \"service-ca-operator-57cdc4b768-lkct8\" (UID: \"08540561-3f4b-46ef-b859-d132f8488e4c\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-lkct8"
Sep 17 17:58:00.904896 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.904854    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-496br\" (UniqueName: \"kubernetes.io/projected/9a99ade3-9dfb-4718-bf9b-07d713715bab-kube-api-access-496br\") pod \"cluster-monitoring-operator-847b8bd8d7-zt5hc\" (UID: \"9a99ade3-9dfb-4718-bf9b-07d713715bab\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zt5hc"
Sep 17 17:58:00.990726 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.990640    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b41e1ab1-9be9-4db0-82da-f3dc6d863a24-serving-cert\") pod \"console-operator-fd4bb46c-l5vq7\" (UID: \"b41e1ab1-9be9-4db0-82da-f3dc6d863a24\") " pod="openshift-console-operator/console-operator-fd4bb46c-l5vq7"
Sep 17 17:58:00.990882 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.990736    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b41e1ab1-9be9-4db0-82da-f3dc6d863a24-trusted-ca\") pod \"console-operator-fd4bb46c-l5vq7\" (UID: \"b41e1ab1-9be9-4db0-82da-f3dc6d863a24\") " pod="openshift-console-operator/console-operator-fd4bb46c-l5vq7"
Sep 17 17:58:00.990882 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.990758    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmsvx\" (UniqueName: \"kubernetes.io/projected/c8209260-6dd4-4d36-a654-d4675c37356b-kube-api-access-wmsvx\") pod \"cluster-samples-operator-685d7d68b5-gw2lt\" (UID: \"c8209260-6dd4-4d36-a654-d4675c37356b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gw2lt"
Sep 17 17:58:00.990882 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.990794    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b41e1ab1-9be9-4db0-82da-f3dc6d863a24-config\") pod \"console-operator-fd4bb46c-l5vq7\" (UID: \"b41e1ab1-9be9-4db0-82da-f3dc6d863a24\") " pod="openshift-console-operator/console-operator-fd4bb46c-l5vq7"
Sep 17 17:58:00.990882 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.990833    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rwlr\" (UniqueName: \"kubernetes.io/projected/b41e1ab1-9be9-4db0-82da-f3dc6d863a24-kube-api-access-9rwlr\") pod \"console-operator-fd4bb46c-l5vq7\" (UID: \"b41e1ab1-9be9-4db0-82da-f3dc6d863a24\") " pod="openshift-console-operator/console-operator-fd4bb46c-l5vq7"
Sep 17 17:58:00.990882 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.990858    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c8209260-6dd4-4d36-a654-d4675c37356b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-gw2lt\" (UID: \"c8209260-6dd4-4d36-a654-d4675c37356b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gw2lt"
Sep 17 17:58:00.991151 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:00.990972    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:58:00.991151 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:00.991037    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8209260-6dd4-4d36-a654-d4675c37356b-samples-operator-tls podName:c8209260-6dd4-4d36-a654-d4675c37356b nodeName:}" failed. No retries permitted until 2026-09-17 17:58:01.49102127 +0000 UTC m=+69.581642897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c8209260-6dd4-4d36-a654-d4675c37356b-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-gw2lt" (UID: "c8209260-6dd4-4d36-a654-d4675c37356b") : secret "samples-operator-tls" not found
Sep 17 17:58:00.991947 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.991922    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b41e1ab1-9be9-4db0-82da-f3dc6d863a24-config\") pod \"console-operator-fd4bb46c-l5vq7\" (UID: \"b41e1ab1-9be9-4db0-82da-f3dc6d863a24\") " pod="openshift-console-operator/console-operator-fd4bb46c-l5vq7"
Sep 17 17:58:00.992547 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.992525    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b41e1ab1-9be9-4db0-82da-f3dc6d863a24-trusted-ca\") pod \"console-operator-fd4bb46c-l5vq7\" (UID: \"b41e1ab1-9be9-4db0-82da-f3dc6d863a24\") " pod="openshift-console-operator/console-operator-fd4bb46c-l5vq7"
Sep 17 17:58:00.993766 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:00.993743    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b41e1ab1-9be9-4db0-82da-f3dc6d863a24-serving-cert\") pod \"console-operator-fd4bb46c-l5vq7\" (UID: \"b41e1ab1-9be9-4db0-82da-f3dc6d863a24\") " pod="openshift-console-operator/console-operator-fd4bb46c-l5vq7"
Sep 17 17:58:01.006982 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:01.006958    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmsvx\" (UniqueName: \"kubernetes.io/projected/c8209260-6dd4-4d36-a654-d4675c37356b-kube-api-access-wmsvx\") pod \"cluster-samples-operator-685d7d68b5-gw2lt\" (UID: \"c8209260-6dd4-4d36-a654-d4675c37356b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gw2lt"
Sep 17 17:58:01.009086 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:01.009062    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rwlr\" (UniqueName: \"kubernetes.io/projected/b41e1ab1-9be9-4db0-82da-f3dc6d863a24-kube-api-access-9rwlr\") pod \"console-operator-fd4bb46c-l5vq7\" (UID: \"b41e1ab1-9be9-4db0-82da-f3dc6d863a24\") " pod="openshift-console-operator/console-operator-fd4bb46c-l5vq7"
Sep 17 17:58:01.034212 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:01.033951    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-lkct8"
Sep 17 17:58:01.138037 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:01.138000    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-l5vq7"
Sep 17 17:58:01.394791 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:01.394712    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a99ade3-9dfb-4718-bf9b-07d713715bab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-zt5hc\" (UID: \"9a99ade3-9dfb-4718-bf9b-07d713715bab\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zt5hc"
Sep 17 17:58:01.394960 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:01.394886    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:58:01.395003 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:01.394963    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a99ade3-9dfb-4718-bf9b-07d713715bab-cluster-monitoring-operator-tls podName:9a99ade3-9dfb-4718-bf9b-07d713715bab nodeName:}" failed. No retries permitted until 2026-09-17 17:58:02.394947718 +0000 UTC m=+70.485569346 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9a99ade3-9dfb-4718-bf9b-07d713715bab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-zt5hc" (UID: "9a99ade3-9dfb-4718-bf9b-07d713715bab") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:58:01.495730 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:01.495694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c8209260-6dd4-4d36-a654-d4675c37356b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-gw2lt\" (UID: \"c8209260-6dd4-4d36-a654-d4675c37356b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gw2lt"
Sep 17 17:58:01.495931 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:01.495860    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:58:01.496005 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:01.495953    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8209260-6dd4-4d36-a654-d4675c37356b-samples-operator-tls podName:c8209260-6dd4-4d36-a654-d4675c37356b nodeName:}" failed. No retries permitted until 2026-09-17 17:58:02.49593598 +0000 UTC m=+70.586557608 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c8209260-6dd4-4d36-a654-d4675c37356b-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-gw2lt" (UID: "c8209260-6dd4-4d36-a654-d4675c37356b") : secret "samples-operator-tls" not found
Sep 17 17:58:02.402045 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:02.402006    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a99ade3-9dfb-4718-bf9b-07d713715bab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-zt5hc\" (UID: \"9a99ade3-9dfb-4718-bf9b-07d713715bab\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zt5hc"
Sep 17 17:58:02.402593 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:02.402141    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:58:02.402593 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:02.402213    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a99ade3-9dfb-4718-bf9b-07d713715bab-cluster-monitoring-operator-tls podName:9a99ade3-9dfb-4718-bf9b-07d713715bab nodeName:}" failed. No retries permitted until 2026-09-17 17:58:04.402191984 +0000 UTC m=+72.492813611 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9a99ade3-9dfb-4718-bf9b-07d713715bab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-zt5hc" (UID: "9a99ade3-9dfb-4718-bf9b-07d713715bab") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:58:02.502965 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:02.502927    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c8209260-6dd4-4d36-a654-d4675c37356b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-gw2lt\" (UID: \"c8209260-6dd4-4d36-a654-d4675c37356b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gw2lt"
Sep 17 17:58:02.503127 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:02.503088    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:58:02.503182 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:02.503171    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8209260-6dd4-4d36-a654-d4675c37356b-samples-operator-tls podName:c8209260-6dd4-4d36-a654-d4675c37356b nodeName:}" failed. No retries permitted until 2026-09-17 17:58:04.503149024 +0000 UTC m=+72.593770653 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c8209260-6dd4-4d36-a654-d4675c37356b-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-gw2lt" (UID: "c8209260-6dd4-4d36-a654-d4675c37356b") : secret "samples-operator-tls" not found
Sep 17 17:58:03.417202 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:03.417129    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-l5vq7"]
Sep 17 17:58:03.419838 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:03.419811    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-lkct8"]
Sep 17 17:58:03.420680 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:58:03.420652    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb41e1ab1_9be9_4db0_82da_f3dc6d863a24.slice/crio-608c43c0300a6f520c46777958c6abbeaa869612e544231710f49e47de87178f WatchSource:0}: Error finding container 608c43c0300a6f520c46777958c6abbeaa869612e544231710f49e47de87178f: Status 404 returned error can't find the container with id 608c43c0300a6f520c46777958c6abbeaa869612e544231710f49e47de87178f
Sep 17 17:58:03.422538 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:58:03.422519    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08540561_3f4b_46ef_b859_d132f8488e4c.slice/crio-723f386a3c753ce51cd26083814fd0733f61db9bb69101a43d6bf8fa47cdc72e WatchSource:0}: Error finding container 723f386a3c753ce51cd26083814fd0733f61db9bb69101a43d6bf8fa47cdc72e: Status 404 returned error can't find the container with id 723f386a3c753ce51cd26083814fd0733f61db9bb69101a43d6bf8fa47cdc72e
Sep 17 17:58:03.704407 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:03.704376    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9xft5" event={"ID":"33a2724c-596d-416e-99e2-583a9db84dcf","Type":"ContainerStarted","Data":"b8271c91eda6e0587d245cb99acaa4890f1cd1eb80fcb4907d9a7ace19dd6cfc"}
Sep 17 17:58:03.705863 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:03.705834    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z7848" event={"ID":"a99a3614-e2dc-4d83-9bed-153654b4bafe","Type":"ContainerStarted","Data":"77aa13573069cd8465686ea300f2a62cd81f512de8f55c9bd70d77008f457679"}
Sep 17 17:58:03.706003 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:03.705924    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-z7848"
Sep 17 17:58:03.706910 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:03.706880    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-lkct8" event={"ID":"08540561-3f4b-46ef-b859-d132f8488e4c","Type":"ContainerStarted","Data":"723f386a3c753ce51cd26083814fd0733f61db9bb69101a43d6bf8fa47cdc72e"}
Sep 17 17:58:03.707846 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:03.707829    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-l5vq7" event={"ID":"b41e1ab1-9be9-4db0-82da-f3dc6d863a24","Type":"ContainerStarted","Data":"608c43c0300a6f520c46777958c6abbeaa869612e544231710f49e47de87178f"}
Sep 17 17:58:03.893599 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:03.893549    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9xft5" podStartSLOduration=66.312180566 podStartE2EDuration="1m10.893531278s" totalImagesPullingTime="4.581350712s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:03.891758663 +0000 UTC m=+71.982380338" watchObservedRunningTime="2026-09-17 17:58:03.893531278 +0000 UTC m=+71.984152919"
Sep 17 17:58:03.942291 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:03.942199    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-z7848" podStartSLOduration=67.689530558 podStartE2EDuration="1m11.942177576s" totalImagesPullingTime="4.252647018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:56:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:03.941730532 +0000 UTC m=+72.032352184" watchObservedRunningTime="2026-09-17 17:58:03.942177576 +0000 UTC m=+72.032799227"
Sep 17 17:58:04.419624 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:04.419583    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a99ade3-9dfb-4718-bf9b-07d713715bab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-zt5hc\" (UID: \"9a99ade3-9dfb-4718-bf9b-07d713715bab\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zt5hc"
Sep 17 17:58:04.420096 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:04.419737    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:58:04.420096 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:04.419797    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a99ade3-9dfb-4718-bf9b-07d713715bab-cluster-monitoring-operator-tls podName:9a99ade3-9dfb-4718-bf9b-07d713715bab nodeName:}" failed. No retries permitted until 2026-09-17 17:58:08.419782614 +0000 UTC m=+76.510404242 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9a99ade3-9dfb-4718-bf9b-07d713715bab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-zt5hc" (UID: "9a99ade3-9dfb-4718-bf9b-07d713715bab") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:58:04.520386 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:04.520353    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c8209260-6dd4-4d36-a654-d4675c37356b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-gw2lt\" (UID: \"c8209260-6dd4-4d36-a654-d4675c37356b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gw2lt"
Sep 17 17:58:04.520564 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:04.520489    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:58:04.520564 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:04.520541    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8209260-6dd4-4d36-a654-d4675c37356b-samples-operator-tls podName:c8209260-6dd4-4d36-a654-d4675c37356b nodeName:}" failed. No retries permitted until 2026-09-17 17:58:08.520525915 +0000 UTC m=+76.611147555 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c8209260-6dd4-4d36-a654-d4675c37356b-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-gw2lt" (UID: "c8209260-6dd4-4d36-a654-d4675c37356b") : secret "samples-operator-tls" not found
Sep 17 17:58:05.111532 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:05.111495    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rw8pw"]
Sep 17 17:58:05.115111 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:05.115087    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rw8pw"
Sep 17 17:58:05.121338 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:05.121313    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 17:58:05.121466 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:05.121374    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 17:58:05.123434 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:05.122575    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 17:58:05.123434 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:05.122742    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 17:58:05.123434 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:05.122763    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ssr9g\""
Sep 17 17:58:05.134317 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:05.134292    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rw8pw"]
Sep 17 17:58:05.226278 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:05.226234    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-crio-socket\") pod \"insights-runtime-extractor-rw8pw\" (UID: \"13b39edd-14cc-42ac-b8e6-e6c61ae9018e\") " pod="openshift-insights/insights-runtime-extractor-rw8pw"
Sep 17 17:58:05.226429 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:05.226284    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rw8pw\" (UID: \"13b39edd-14cc-42ac-b8e6-e6c61ae9018e\") " pod="openshift-insights/insights-runtime-extractor-rw8pw"
Sep 17 17:58:05.226429 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:05.226410    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-data-volume\") pod \"insights-runtime-extractor-rw8pw\" (UID: \"13b39edd-14cc-42ac-b8e6-e6c61ae9018e\") " pod="openshift-insights/insights-runtime-extractor-rw8pw"
Sep 17 17:58:05.226510 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:05.226437    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rw8pw\" (UID: \"13b39edd-14cc-42ac-b8e6-e6c61ae9018e\") " pod="openshift-insights/insights-runtime-extractor-rw8pw"
Sep 17 17:58:05.226510 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:05.226461    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhztk\" (UniqueName: \"kubernetes.io/projected/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-kube-api-access-mhztk\") pod \"insights-runtime-extractor-rw8pw\" (UID: \"13b39edd-14cc-42ac-b8e6-e6c61ae9018e\") " pod="openshift-insights/insights-runtime-extractor-rw8pw"
Sep 17 17:58:05.327723 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:05.327689    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhztk\" (UniqueName: \"kubernetes.io/projected/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-kube-api-access-mhztk\") pod \"insights-runtime-extractor-rw8pw\" (UID: \"13b39edd-14cc-42ac-b8e6-e6c61ae9018e\") " pod="openshift-insights/insights-runtime-extractor-rw8pw"
Sep 17 17:58:05.327932 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:05.327769    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-crio-socket\") pod \"insights-runtime-extractor-rw8pw\" (UID: \"13b39edd-14cc-42ac-b8e6-e6c61ae9018e\") " pod="openshift-insights/insights-runtime-extractor-rw8pw"
Sep 17 17:58:05.327932 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:05.327792    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rw8pw\" (UID: \"13b39edd-14cc-42ac-b8e6-e6c61ae9018e\") " pod="openshift-insights/insights-runtime-extractor-rw8pw"
Sep 17 17:58:05.327932 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:05.327851    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-data-volume\") pod \"insights-runtime-extractor-rw8pw\" (UID: \"13b39edd-14cc-42ac-b8e6-e6c61ae9018e\") " pod="openshift-insights/insights-runtime-extractor-rw8pw"
Sep 17 17:58:05.327932 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:05.327872    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rw8pw\" (UID: \"13b39edd-14cc-42ac-b8e6-e6c61ae9018e\") " pod="openshift-insights/insights-runtime-extractor-rw8pw"
Sep 17 17:58:05.327932 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:05.327915    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-crio-socket\") pod \"insights-runtime-extractor-rw8pw\" (UID: \"13b39edd-14cc-42ac-b8e6-e6c61ae9018e\") " pod="openshift-insights/insights-runtime-extractor-rw8pw"
Sep 17 17:58:05.328201 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:05.327982    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:58:05.328201 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:05.327981    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:05.328201 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:05.328046    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-kube-rbac-proxy-cm podName:13b39edd-14cc-42ac-b8e6-e6c61ae9018e nodeName:}" failed. No retries permitted until 2026-09-17 17:58:05.828030944 +0000 UTC m=+73.918652594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rw8pw" (UID: "13b39edd-14cc-42ac-b8e6-e6c61ae9018e") : configmap "kube-rbac-proxy" not found
Sep 17 17:58:05.328201 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:05.328094    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-insights-runtime-extractor-tls podName:13b39edd-14cc-42ac-b8e6-e6c61ae9018e nodeName:}" failed. No retries permitted until 2026-09-17 17:58:05.828074317 +0000 UTC m=+73.918695957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rw8pw" (UID: "13b39edd-14cc-42ac-b8e6-e6c61ae9018e") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:05.328429 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:05.328302    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-data-volume\") pod \"insights-runtime-extractor-rw8pw\" (UID: \"13b39edd-14cc-42ac-b8e6-e6c61ae9018e\") " pod="openshift-insights/insights-runtime-extractor-rw8pw"
Sep 17 17:58:05.337546 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:05.337517    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhztk\" (UniqueName: \"kubernetes.io/projected/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-kube-api-access-mhztk\") pod \"insights-runtime-extractor-rw8pw\" (UID: \"13b39edd-14cc-42ac-b8e6-e6c61ae9018e\") " pod="openshift-insights/insights-runtime-extractor-rw8pw"
Sep 17 17:58:05.832846 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:05.832813    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rw8pw\" (UID: \"13b39edd-14cc-42ac-b8e6-e6c61ae9018e\") " pod="openshift-insights/insights-runtime-extractor-rw8pw"
Sep 17 17:58:05.833191 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:05.832894    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rw8pw\" (UID: \"13b39edd-14cc-42ac-b8e6-e6c61ae9018e\") " pod="openshift-insights/insights-runtime-extractor-rw8pw"
Sep 17 17:58:05.833191 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:05.832979    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:58:05.833191 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:05.833018    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:05.833191 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:05.833052    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-kube-rbac-proxy-cm podName:13b39edd-14cc-42ac-b8e6-e6c61ae9018e nodeName:}" failed. No retries permitted until 2026-09-17 17:58:06.833037224 +0000 UTC m=+74.923658875 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rw8pw" (UID: "13b39edd-14cc-42ac-b8e6-e6c61ae9018e") : configmap "kube-rbac-proxy" not found
Sep 17 17:58:05.833191 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:05.833094    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-insights-runtime-extractor-tls podName:13b39edd-14cc-42ac-b8e6-e6c61ae9018e nodeName:}" failed. No retries permitted until 2026-09-17 17:58:06.833076649 +0000 UTC m=+74.923698291 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rw8pw" (UID: "13b39edd-14cc-42ac-b8e6-e6c61ae9018e") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:06.716210 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:06.716159    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-lkct8" event={"ID":"08540561-3f4b-46ef-b859-d132f8488e4c","Type":"ContainerStarted","Data":"f499eb3b221b94fa2d48aa9d24dd178ce22ec7e110c6b9c84471205991eaa582"}
Sep 17 17:58:06.717815 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:06.717793    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/0.log"
Sep 17 17:58:06.717945 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:06.717829    2581 generic.go:356] "Generic (PLEG): container finished" podID="b41e1ab1-9be9-4db0-82da-f3dc6d863a24" containerID="e6dd0eef86327b2a7ac9334927dacd37b0f42fda1afa866a43562482038d03bf" exitCode=255
Sep 17 17:58:06.717945 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:06.717878    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-l5vq7" event={"ID":"b41e1ab1-9be9-4db0-82da-f3dc6d863a24","Type":"ContainerDied","Data":"e6dd0eef86327b2a7ac9334927dacd37b0f42fda1afa866a43562482038d03bf"}
Sep 17 17:58:06.718080 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:06.718068    2581 scope.go:117] "RemoveContainer" containerID="e6dd0eef86327b2a7ac9334927dacd37b0f42fda1afa866a43562482038d03bf"
Sep 17 17:58:06.734111 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:06.734064    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-lkct8" podStartSLOduration=4.404617476 podStartE2EDuration="6.734044549s" totalImagesPullingTime="2.329427073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:06.733644228 +0000 UTC m=+74.824265882" watchObservedRunningTime="2026-09-17 17:58:06.734044549 +0000 UTC m=+74.824666200"
Sep 17 17:58:06.840517 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:06.840494    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rw8pw\" (UID: \"13b39edd-14cc-42ac-b8e6-e6c61ae9018e\") " pod="openshift-insights/insights-runtime-extractor-rw8pw"
Sep 17 17:58:06.840889 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:06.840627    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:58:06.840889 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:06.840698    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-kube-rbac-proxy-cm podName:13b39edd-14cc-42ac-b8e6-e6c61ae9018e nodeName:}" failed. No retries permitted until 2026-09-17 17:58:08.840679021 +0000 UTC m=+76.931300670 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rw8pw" (UID: "13b39edd-14cc-42ac-b8e6-e6c61ae9018e") : configmap "kube-rbac-proxy" not found
Sep 17 17:58:06.841045 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:06.841008    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rw8pw\" (UID: \"13b39edd-14cc-42ac-b8e6-e6c61ae9018e\") " pod="openshift-insights/insights-runtime-extractor-rw8pw"
Sep 17 17:58:06.841139 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:06.841111    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:06.841238 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:06.841168    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-insights-runtime-extractor-tls podName:13b39edd-14cc-42ac-b8e6-e6c61ae9018e nodeName:}" failed. No retries permitted until 2026-09-17 17:58:08.841152728 +0000 UTC m=+76.931774356 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rw8pw" (UID: "13b39edd-14cc-42ac-b8e6-e6c61ae9018e") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:07.722059 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:07.722026    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/1.log"
Sep 17 17:58:07.722443 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:07.722424    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/0.log"
Sep 17 17:58:07.722499 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:07.722461    2581 generic.go:356] "Generic (PLEG): container finished" podID="b41e1ab1-9be9-4db0-82da-f3dc6d863a24" containerID="d1937559867f807d91b7f7adad75981b15c202ddbb97b08d6b6a78757d4f1cc1" exitCode=255
Sep 17 17:58:07.722531 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:07.722495    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-l5vq7" event={"ID":"b41e1ab1-9be9-4db0-82da-f3dc6d863a24","Type":"ContainerDied","Data":"d1937559867f807d91b7f7adad75981b15c202ddbb97b08d6b6a78757d4f1cc1"}
Sep 17 17:58:07.722563 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:07.722557    2581 scope.go:117] "RemoveContainer" containerID="e6dd0eef86327b2a7ac9334927dacd37b0f42fda1afa866a43562482038d03bf"
Sep 17 17:58:07.722746 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:07.722724    2581 scope.go:117] "RemoveContainer" containerID="d1937559867f807d91b7f7adad75981b15c202ddbb97b08d6b6a78757d4f1cc1"
Sep 17 17:58:07.722951 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:07.722930    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-l5vq7_openshift-console-operator(b41e1ab1-9be9-4db0-82da-f3dc6d863a24)\"" pod="openshift-console-operator/console-operator-fd4bb46c-l5vq7" podUID="b41e1ab1-9be9-4db0-82da-f3dc6d863a24"
Sep 17 17:58:07.827048 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:07.827025    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qcqn9_e73ac43a-8a55-40ea-902c-5b06dfcc5d0b/dns-node-resolver/0.log"
Sep 17 17:58:08.454504 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:08.454448    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a99ade3-9dfb-4718-bf9b-07d713715bab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-zt5hc\" (UID: \"9a99ade3-9dfb-4718-bf9b-07d713715bab\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zt5hc"
Sep 17 17:58:08.454854 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:08.454589    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:58:08.454854 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:08.454657    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a99ade3-9dfb-4718-bf9b-07d713715bab-cluster-monitoring-operator-tls podName:9a99ade3-9dfb-4718-bf9b-07d713715bab nodeName:}" failed. No retries permitted until 2026-09-17 17:58:16.454640569 +0000 UTC m=+84.545262198 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9a99ade3-9dfb-4718-bf9b-07d713715bab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-zt5hc" (UID: "9a99ade3-9dfb-4718-bf9b-07d713715bab") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:58:08.555412 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:08.555367    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c8209260-6dd4-4d36-a654-d4675c37356b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-gw2lt\" (UID: \"c8209260-6dd4-4d36-a654-d4675c37356b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gw2lt"
Sep 17 17:58:08.555575 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:08.555538    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 17:58:08.555623 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:08.555611    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8209260-6dd4-4d36-a654-d4675c37356b-samples-operator-tls podName:c8209260-6dd4-4d36-a654-d4675c37356b nodeName:}" failed. No retries permitted until 2026-09-17 17:58:16.555595949 +0000 UTC m=+84.646217577 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c8209260-6dd4-4d36-a654-d4675c37356b-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-gw2lt" (UID: "c8209260-6dd4-4d36-a654-d4675c37356b") : secret "samples-operator-tls" not found
Sep 17 17:58:08.726783 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:08.726700    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/1.log"
Sep 17 17:58:08.727111 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:08.727093    2581 scope.go:117] "RemoveContainer" containerID="d1937559867f807d91b7f7adad75981b15c202ddbb97b08d6b6a78757d4f1cc1"
Sep 17 17:58:08.727296 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:08.727273    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-l5vq7_openshift-console-operator(b41e1ab1-9be9-4db0-82da-f3dc6d863a24)\"" pod="openshift-console-operator/console-operator-fd4bb46c-l5vq7" podUID="b41e1ab1-9be9-4db0-82da-f3dc6d863a24"
Sep 17 17:58:08.857672 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:08.857630    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rw8pw\" (UID: \"13b39edd-14cc-42ac-b8e6-e6c61ae9018e\") " pod="openshift-insights/insights-runtime-extractor-rw8pw"
Sep 17 17:58:08.857876 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:08.857720    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rw8pw\" (UID: \"13b39edd-14cc-42ac-b8e6-e6c61ae9018e\") " pod="openshift-insights/insights-runtime-extractor-rw8pw"
Sep 17 17:58:08.857876 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:08.857783    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:08.857876 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:08.857798    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:58:08.857876 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:08.857854    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-insights-runtime-extractor-tls podName:13b39edd-14cc-42ac-b8e6-e6c61ae9018e nodeName:}" failed. No retries permitted until 2026-09-17 17:58:12.857834901 +0000 UTC m=+80.948456542 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rw8pw" (UID: "13b39edd-14cc-42ac-b8e6-e6c61ae9018e") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:08.857876 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:08.857876    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-kube-rbac-proxy-cm podName:13b39edd-14cc-42ac-b8e6-e6c61ae9018e nodeName:}" failed. No retries permitted until 2026-09-17 17:58:12.857866355 +0000 UTC m=+80.948487986 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rw8pw" (UID: "13b39edd-14cc-42ac-b8e6-e6c61ae9018e") : configmap "kube-rbac-proxy" not found
Sep 17 17:58:08.986280 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:08.986202    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zsz9h_4383fe04-b7e0-4dd9-b3cc-ddca3d57aed2/node-ca/0.log"
Sep 17 17:58:09.385169 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:09.385094    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6nv7n_06dbe6ba-8122-4c23-9dbe-63e2de449f7c/serve-healthcheck-canary/0.log"
Sep 17 17:58:09.409250 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:09.409218    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-85bx6"]
Sep 17 17:58:09.413289 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:09.413274    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-85bx6"
Sep 17 17:58:09.416616 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:09.416595    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 17 17:58:09.417649 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:09.417627    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 17 17:58:09.417649 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:09.417644    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 17 17:58:09.417804 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:09.417697    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-cq62x\""
Sep 17 17:58:09.417950 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:09.417936    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 17 17:58:09.423429 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:09.423409    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-85bx6"]
Sep 17 17:58:09.462560 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:09.462529    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7cd6bb99-033d-483d-9dda-8e7b5e50638a-signing-key\") pod \"service-ca-55f6fc7b86-85bx6\" (UID: \"7cd6bb99-033d-483d-9dda-8e7b5e50638a\") " pod="openshift-service-ca/service-ca-55f6fc7b86-85bx6"
Sep 17 17:58:09.462560 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:09.462563    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7wsj\" (UniqueName: \"kubernetes.io/projected/7cd6bb99-033d-483d-9dda-8e7b5e50638a-kube-api-access-n7wsj\") pod \"service-ca-55f6fc7b86-85bx6\" (UID: \"7cd6bb99-033d-483d-9dda-8e7b5e50638a\") " pod="openshift-service-ca/service-ca-55f6fc7b86-85bx6"
Sep 17 17:58:09.462936 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:09.462632    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7cd6bb99-033d-483d-9dda-8e7b5e50638a-signing-cabundle\") pod \"service-ca-55f6fc7b86-85bx6\" (UID: \"7cd6bb99-033d-483d-9dda-8e7b5e50638a\") " pod="openshift-service-ca/service-ca-55f6fc7b86-85bx6"
Sep 17 17:58:09.564012 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:09.563975    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7cd6bb99-033d-483d-9dda-8e7b5e50638a-signing-cabundle\") pod \"service-ca-55f6fc7b86-85bx6\" (UID: \"7cd6bb99-033d-483d-9dda-8e7b5e50638a\") " pod="openshift-service-ca/service-ca-55f6fc7b86-85bx6"
Sep 17 17:58:09.564178 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:09.564102    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7cd6bb99-033d-483d-9dda-8e7b5e50638a-signing-key\") pod \"service-ca-55f6fc7b86-85bx6\" (UID: \"7cd6bb99-033d-483d-9dda-8e7b5e50638a\") " pod="openshift-service-ca/service-ca-55f6fc7b86-85bx6"
Sep 17 17:58:09.564178 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:09.564132    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7wsj\" (UniqueName: \"kubernetes.io/projected/7cd6bb99-033d-483d-9dda-8e7b5e50638a-kube-api-access-n7wsj\") pod \"service-ca-55f6fc7b86-85bx6\" (UID: \"7cd6bb99-033d-483d-9dda-8e7b5e50638a\") " pod="openshift-service-ca/service-ca-55f6fc7b86-85bx6"
Sep 17 17:58:09.564614 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:09.564593    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7cd6bb99-033d-483d-9dda-8e7b5e50638a-signing-cabundle\") pod \"service-ca-55f6fc7b86-85bx6\" (UID: \"7cd6bb99-033d-483d-9dda-8e7b5e50638a\") " pod="openshift-service-ca/service-ca-55f6fc7b86-85bx6"
Sep 17 17:58:09.566444 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:09.566425    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7cd6bb99-033d-483d-9dda-8e7b5e50638a-signing-key\") pod \"service-ca-55f6fc7b86-85bx6\" (UID: \"7cd6bb99-033d-483d-9dda-8e7b5e50638a\") " pod="openshift-service-ca/service-ca-55f6fc7b86-85bx6"
Sep 17 17:58:09.574160 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:09.574140    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7wsj\" (UniqueName: \"kubernetes.io/projected/7cd6bb99-033d-483d-9dda-8e7b5e50638a-kube-api-access-n7wsj\") pod \"service-ca-55f6fc7b86-85bx6\" (UID: \"7cd6bb99-033d-483d-9dda-8e7b5e50638a\") " pod="openshift-service-ca/service-ca-55f6fc7b86-85bx6"
Sep 17 17:58:09.722125 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:09.722096    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-85bx6"
Sep 17 17:58:09.998005 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:09.997931    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-85bx6"]
Sep 17 17:58:10.001377 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:58:10.001355    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd6bb99_033d_483d_9dda_8e7b5e50638a.slice/crio-9567419411fa24c783e79f9e18f41294117588a66d404e88219efc55be2bf5b7 WatchSource:0}: Error finding container 9567419411fa24c783e79f9e18f41294117588a66d404e88219efc55be2bf5b7: Status 404 returned error can't find the container with id 9567419411fa24c783e79f9e18f41294117588a66d404e88219efc55be2bf5b7
Sep 17 17:58:10.732735 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:10.732702    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-85bx6" event={"ID":"7cd6bb99-033d-483d-9dda-8e7b5e50638a","Type":"ContainerStarted","Data":"9699e5383bfbbff40061d91b1e38afc45325aba603c2d2b3f908539f7a8a215c"}
Sep 17 17:58:10.733155 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:10.732748    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-85bx6" event={"ID":"7cd6bb99-033d-483d-9dda-8e7b5e50638a","Type":"ContainerStarted","Data":"9567419411fa24c783e79f9e18f41294117588a66d404e88219efc55be2bf5b7"}
Sep 17 17:58:10.766475 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:10.766418    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-85bx6" podStartSLOduration=1.7663999320000001 podStartE2EDuration="1.766399932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:10.765993296 +0000 UTC m=+78.856614947" watchObservedRunningTime="2026-09-17 17:58:10.766399932 +0000 UTC m=+78.857021581"
Sep 17 17:58:11.138748 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:11.138666    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-l5vq7"
Sep 17 17:58:11.138748 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:11.138725    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-l5vq7"
Sep 17 17:58:11.139106 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:11.139091    2581 scope.go:117] "RemoveContainer" containerID="d1937559867f807d91b7f7adad75981b15c202ddbb97b08d6b6a78757d4f1cc1"
Sep 17 17:58:11.139284 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:11.139265    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-l5vq7_openshift-console-operator(b41e1ab1-9be9-4db0-82da-f3dc6d863a24)\"" pod="openshift-console-operator/console-operator-fd4bb46c-l5vq7" podUID="b41e1ab1-9be9-4db0-82da-f3dc6d863a24"
Sep 17 17:58:12.894520 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:12.894479    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rw8pw\" (UID: \"13b39edd-14cc-42ac-b8e6-e6c61ae9018e\") " pod="openshift-insights/insights-runtime-extractor-rw8pw"
Sep 17 17:58:12.895009 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:12.894563    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rw8pw\" (UID: \"13b39edd-14cc-42ac-b8e6-e6c61ae9018e\") " pod="openshift-insights/insights-runtime-extractor-rw8pw"
Sep 17 17:58:12.895009 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:12.894587    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:58:12.895009 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:12.894655    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-kube-rbac-proxy-cm podName:13b39edd-14cc-42ac-b8e6-e6c61ae9018e nodeName:}" failed. No retries permitted until 2026-09-17 17:58:20.894639744 +0000 UTC m=+88.985261377 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rw8pw" (UID: "13b39edd-14cc-42ac-b8e6-e6c61ae9018e") : configmap "kube-rbac-proxy" not found
Sep 17 17:58:12.895009 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:12.894673    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:12.895009 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:12.894723    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-insights-runtime-extractor-tls podName:13b39edd-14cc-42ac-b8e6-e6c61ae9018e nodeName:}" failed. No retries permitted until 2026-09-17 17:58:20.894700822 +0000 UTC m=+88.985322463 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rw8pw" (UID: "13b39edd-14cc-42ac-b8e6-e6c61ae9018e") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:16.523956 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:16.523787    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a99ade3-9dfb-4718-bf9b-07d713715bab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-zt5hc\" (UID: \"9a99ade3-9dfb-4718-bf9b-07d713715bab\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zt5hc"
Sep 17 17:58:16.524420 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:16.523955    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:58:16.524420 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:16.524025    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a99ade3-9dfb-4718-bf9b-07d713715bab-cluster-monitoring-operator-tls podName:9a99ade3-9dfb-4718-bf9b-07d713715bab nodeName:}" failed. No retries permitted until 2026-09-17 17:58:32.52400927 +0000 UTC m=+100.614630903 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9a99ade3-9dfb-4718-bf9b-07d713715bab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-zt5hc" (UID: "9a99ade3-9dfb-4718-bf9b-07d713715bab") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:58:16.625007 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:16.624970    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c8209260-6dd4-4d36-a654-d4675c37356b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-gw2lt\" (UID: \"c8209260-6dd4-4d36-a654-d4675c37356b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gw2lt"
Sep 17 17:58:16.627287 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:16.627268    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c8209260-6dd4-4d36-a654-d4675c37356b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-gw2lt\" (UID: \"c8209260-6dd4-4d36-a654-d4675c37356b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gw2lt"
Sep 17 17:58:16.731744 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:16.731706    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gw2lt"
Sep 17 17:58:17.006214 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:17.006191    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gw2lt"]
Sep 17 17:58:17.756518 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:17.756446    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gw2lt" event={"ID":"c8209260-6dd4-4d36-a654-d4675c37356b","Type":"ContainerStarted","Data":"ec5e03b4cca9a4b83cdf7312b5042e93a86d5ecdaaa3c8810d922e1d0cf1cd3c"}
Sep 17 17:58:18.761603 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:18.761568    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gw2lt" event={"ID":"c8209260-6dd4-4d36-a654-d4675c37356b","Type":"ContainerStarted","Data":"5afab05246d7472373065d353d44c98d23ee1e545f9d69f3bfd962aea30e0cc9"}
Sep 17 17:58:18.761994 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:18.761622    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gw2lt" event={"ID":"c8209260-6dd4-4d36-a654-d4675c37356b","Type":"ContainerStarted","Data":"d002a1822d9e572ad1aca2422ea4992599fc59369d86ebb3d13de1f515cc9903"}
Sep 17 17:58:18.797637 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:18.797519    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-gw2lt" podStartSLOduration=17.31497896 podStartE2EDuration="18.7974982s" totalImagesPullingTime="1.48251924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:18.7971211 +0000 UTC m=+86.887742760" watchObservedRunningTime="2026-09-17 17:58:18.7974982 +0000 UTC m=+86.888119851"
Sep 17 17:58:20.962055 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:20.962004    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rw8pw\" (UID: \"13b39edd-14cc-42ac-b8e6-e6c61ae9018e\") " pod="openshift-insights/insights-runtime-extractor-rw8pw"
Sep 17 17:58:20.962444 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:20.962087    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rw8pw\" (UID: \"13b39edd-14cc-42ac-b8e6-e6c61ae9018e\") " pod="openshift-insights/insights-runtime-extractor-rw8pw"
Sep 17 17:58:20.962444 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:20.962177    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:20.962444 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:20.962229    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-insights-runtime-extractor-tls podName:13b39edd-14cc-42ac-b8e6-e6c61ae9018e nodeName:}" failed. No retries permitted until 2026-09-17 17:58:36.962215135 +0000 UTC m=+105.052836763 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rw8pw" (UID: "13b39edd-14cc-42ac-b8e6-e6c61ae9018e") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:58:20.962566 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:20.962545    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rw8pw\" (UID: \"13b39edd-14cc-42ac-b8e6-e6c61ae9018e\") " pod="openshift-insights/insights-runtime-extractor-rw8pw"
Sep 17 17:58:25.352982 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:25.352952    2581 scope.go:117] "RemoveContainer" containerID="d1937559867f807d91b7f7adad75981b15c202ddbb97b08d6b6a78757d4f1cc1"
Sep 17 17:58:25.784256 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:25.784229    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/2.log"
Sep 17 17:58:25.784668 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:25.784654    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/1.log"
Sep 17 17:58:25.784715 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:25.784684    2581 generic.go:356] "Generic (PLEG): container finished" podID="b41e1ab1-9be9-4db0-82da-f3dc6d863a24" containerID="25adf070ef000511fdf0bcfa6dbc92471aecf0754f82910c49da328a4e0689f8" exitCode=255
Sep 17 17:58:25.784748 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:25.784733    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-l5vq7" event={"ID":"b41e1ab1-9be9-4db0-82da-f3dc6d863a24","Type":"ContainerDied","Data":"25adf070ef000511fdf0bcfa6dbc92471aecf0754f82910c49da328a4e0689f8"}
Sep 17 17:58:25.784811 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:25.784800    2581 scope.go:117] "RemoveContainer" containerID="d1937559867f807d91b7f7adad75981b15c202ddbb97b08d6b6a78757d4f1cc1"
Sep 17 17:58:25.785133 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:25.785108    2581 scope.go:117] "RemoveContainer" containerID="25adf070ef000511fdf0bcfa6dbc92471aecf0754f82910c49da328a4e0689f8"
Sep 17 17:58:25.785323 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:25.785300    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-l5vq7_openshift-console-operator(b41e1ab1-9be9-4db0-82da-f3dc6d863a24)\"" pod="openshift-console-operator/console-operator-fd4bb46c-l5vq7" podUID="b41e1ab1-9be9-4db0-82da-f3dc6d863a24"
Sep 17 17:58:26.789344 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:26.789318    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/2.log"
Sep 17 17:58:31.138395 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:31.138362    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-l5vq7"
Sep 17 17:58:31.138821 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:31.138415    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-l5vq7"
Sep 17 17:58:31.138821 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:31.138688    2581 scope.go:117] "RemoveContainer" containerID="25adf070ef000511fdf0bcfa6dbc92471aecf0754f82910c49da328a4e0689f8"
Sep 17 17:58:31.138889 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:31.138826    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-l5vq7_openshift-console-operator(b41e1ab1-9be9-4db0-82da-f3dc6d863a24)\"" pod="openshift-console-operator/console-operator-fd4bb46c-l5vq7" podUID="b41e1ab1-9be9-4db0-82da-f3dc6d863a24"
Sep 17 17:58:31.241010 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:31.240977    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13c59508-5d4e-4ea8-a211-42ec8c5bfffa-metrics-tls\") pod \"dns-default-sgwxz\" (UID: \"13c59508-5d4e-4ea8-a211-42ec8c5bfffa\") " pod="openshift-dns/dns-default-sgwxz"
Sep 17 17:58:31.243340 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:31.243314    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13c59508-5d4e-4ea8-a211-42ec8c5bfffa-metrics-tls\") pod \"dns-default-sgwxz\" (UID: \"13c59508-5d4e-4ea8-a211-42ec8c5bfffa\") " pod="openshift-dns/dns-default-sgwxz"
Sep 17 17:58:31.492101 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:31.492072    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4ffrq\""
Sep 17 17:58:31.499422 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:31.499402    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sgwxz"
Sep 17 17:58:31.774697 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:31.774672    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sgwxz"]
Sep 17 17:58:31.776158 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:58:31.776127    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13c59508_5d4e_4ea8_a211_42ec8c5bfffa.slice/crio-2bba52c203217dac38eaf9d9b29ba2fd20a4078418c79a75392431a7ccfe5bb2 WatchSource:0}: Error finding container 2bba52c203217dac38eaf9d9b29ba2fd20a4078418c79a75392431a7ccfe5bb2: Status 404 returned error can't find the container with id 2bba52c203217dac38eaf9d9b29ba2fd20a4078418c79a75392431a7ccfe5bb2
Sep 17 17:58:31.805292 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:31.805264    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sgwxz" event={"ID":"13c59508-5d4e-4ea8-a211-42ec8c5bfffa","Type":"ContainerStarted","Data":"2bba52c203217dac38eaf9d9b29ba2fd20a4078418c79a75392431a7ccfe5bb2"}
Sep 17 17:58:32.550076 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:32.550041    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a99ade3-9dfb-4718-bf9b-07d713715bab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-zt5hc\" (UID: \"9a99ade3-9dfb-4718-bf9b-07d713715bab\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zt5hc"
Sep 17 17:58:32.552701 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:32.552668    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a99ade3-9dfb-4718-bf9b-07d713715bab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-zt5hc\" (UID: \"9a99ade3-9dfb-4718-bf9b-07d713715bab\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zt5hc"
Sep 17 17:58:32.824964 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:32.824875    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zt5hc"
Sep 17 17:58:33.439254 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:33.439231    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zt5hc"]
Sep 17 17:58:33.441590 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:58:33.441566    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a99ade3_9dfb_4718_bf9b_07d713715bab.slice/crio-cac4a836a5bf479e7b738e2640c6075813502edc6844ce4ea14460e1c6b9e08f WatchSource:0}: Error finding container cac4a836a5bf479e7b738e2640c6075813502edc6844ce4ea14460e1c6b9e08f: Status 404 returned error can't find the container with id cac4a836a5bf479e7b738e2640c6075813502edc6844ce4ea14460e1c6b9e08f
Sep 17 17:58:33.813687 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:33.813656    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sgwxz" event={"ID":"13c59508-5d4e-4ea8-a211-42ec8c5bfffa","Type":"ContainerStarted","Data":"5a625c54987e21a2b4428704b530cd6cd02c9227a253b623c4ff756bcdd19b2d"}
Sep 17 17:58:33.814183 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:33.813706    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sgwxz" event={"ID":"13c59508-5d4e-4ea8-a211-42ec8c5bfffa","Type":"ContainerStarted","Data":"84433a3671a829d68564ce912a8f88839912db69bfc94978b7b4c54fd14fd528"}
Sep 17 17:58:33.814183 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:33.813755    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sgwxz"
Sep 17 17:58:33.814896 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:33.814876    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zt5hc" event={"ID":"9a99ade3-9dfb-4718-bf9b-07d713715bab","Type":"ContainerStarted","Data":"cac4a836a5bf479e7b738e2640c6075813502edc6844ce4ea14460e1c6b9e08f"}
Sep 17 17:58:33.840159 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:33.840116    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-sgwxz" podStartSLOduration=65.40651673 podStartE2EDuration="1m6.840100544s" totalImagesPullingTime="1.433583814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:33.838507594 +0000 UTC m=+101.929129270" watchObservedRunningTime="2026-09-17 17:58:33.840100544 +0000 UTC m=+101.930722194"
Sep 17 17:58:34.712428 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:34.712394    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-z7848"
Sep 17 17:58:35.823518 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:35.823489    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zt5hc" event={"ID":"9a99ade3-9dfb-4718-bf9b-07d713715bab","Type":"ContainerStarted","Data":"ff1c11a82df349222d8cab022c90f562c20aafe53f41c0314136459b1db3064e"}
Sep 17 17:58:35.842767 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:35.842726    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-zt5hc" podStartSLOduration=34.300212855 podStartE2EDuration="35.842708988s" totalImagesPullingTime="1.542496133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:35.84264824 +0000 UTC m=+103.933269893" watchObservedRunningTime="2026-09-17 17:58:35.842708988 +0000 UTC m=+103.933330637"
Sep 17 17:58:36.986948 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:36.986895    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rw8pw\" (UID: \"13b39edd-14cc-42ac-b8e6-e6c61ae9018e\") " pod="openshift-insights/insights-runtime-extractor-rw8pw"
Sep 17 17:58:36.989322 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:36.989289    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/13b39edd-14cc-42ac-b8e6-e6c61ae9018e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rw8pw\" (UID: \"13b39edd-14cc-42ac-b8e6-e6c61ae9018e\") " pod="openshift-insights/insights-runtime-extractor-rw8pw"
Sep 17 17:58:37.227974 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:37.227932    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rw8pw"
Sep 17 17:58:37.533109 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:37.533033    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rw8pw"]
Sep 17 17:58:37.535964 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:58:37.535936    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13b39edd_14cc_42ac_b8e6_e6c61ae9018e.slice/crio-d5cb36359c2d18f88d3c7f321791d218bad93204007320362b4b382b5e3c4290 WatchSource:0}: Error finding container d5cb36359c2d18f88d3c7f321791d218bad93204007320362b4b382b5e3c4290: Status 404 returned error can't find the container with id d5cb36359c2d18f88d3c7f321791d218bad93204007320362b4b382b5e3c4290
Sep 17 17:58:37.831121 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:37.831042    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rw8pw" event={"ID":"13b39edd-14cc-42ac-b8e6-e6c61ae9018e","Type":"ContainerStarted","Data":"a55c18fb25aa0750deaf0661ad561a6a5927df3f7ee7237f32b802c39a9fb2d9"}
Sep 17 17:58:37.831121 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:37.831097    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rw8pw" event={"ID":"13b39edd-14cc-42ac-b8e6-e6c61ae9018e","Type":"ContainerStarted","Data":"d5cb36359c2d18f88d3c7f321791d218bad93204007320362b4b382b5e3c4290"}
Sep 17 17:58:39.839362 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:39.839330    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rw8pw" event={"ID":"13b39edd-14cc-42ac-b8e6-e6c61ae9018e","Type":"ContainerStarted","Data":"82230f7afe7a563caf518dd224addfcd37a423dac34fe99a68fb9267b6cd3b7e"}
Sep 17 17:58:40.845229 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:40.845143    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rw8pw" event={"ID":"13b39edd-14cc-42ac-b8e6-e6c61ae9018e","Type":"ContainerStarted","Data":"77fe48282426605fb35a215c58a7abc19454b704ddb8e388737855368ee5a75e"}
Sep 17 17:58:40.875510 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:40.875455    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rw8pw" podStartSLOduration=33.090509266 podStartE2EDuration="35.875431653s" totalImagesPullingTime="2.784922387s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:58:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:40.874921926 +0000 UTC m=+108.965543570" watchObservedRunningTime="2026-09-17 17:58:40.875431653 +0000 UTC m=+108.966053303"
Sep 17 17:58:43.821562 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:43.821528    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sgwxz"
Sep 17 17:58:44.352504 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:44.352467    2581 scope.go:117] "RemoveContainer" containerID="25adf070ef000511fdf0bcfa6dbc92471aecf0754f82910c49da328a4e0689f8"
Sep 17 17:58:44.352694 ip-10-0-130-28 kubenswrapper[2581]: E0917 17:58:44.352650    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-l5vq7_openshift-console-operator(b41e1ab1-9be9-4db0-82da-f3dc6d863a24)\"" pod="openshift-console-operator/console-operator-fd4bb46c-l5vq7" podUID="b41e1ab1-9be9-4db0-82da-f3dc6d863a24"
Sep 17 17:58:46.693674 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.693629    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-2gv9q"]
Sep 17 17:58:46.695658 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.695641    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2gv9q"
Sep 17 17:58:46.698752 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.698731    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 17 17:58:46.699339 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.699314    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-bzrsk\""
Sep 17 17:58:46.700347 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.700326    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 17:58:46.705463 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.705444    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 17 17:58:46.734077 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.734048    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-7tlp8"]
Sep 17 17:58:46.736112 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.736090    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-2gv9q"]
Sep 17 17:58:46.736249 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.736198    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-7tlp8"
Sep 17 17:58:46.739730 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.739704    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 17:58:46.739979 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.739965    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-xnnxx\""
Sep 17 17:58:46.740496 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.740480    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 17:58:46.743675 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.743650    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 17:58:46.770045 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.770014    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-7tlp8"]
Sep 17 17:58:46.773285 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.773259    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5fr6x"]
Sep 17 17:58:46.776111 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.776091    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5fr6x"
Sep 17 17:58:46.786555 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.786532    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 17:58:46.787135 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.787113    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 17:58:46.787225 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.787150    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2w4t8\""
Sep 17 17:58:46.792703 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.792687    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 17:58:46.860865 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.860831    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdc2b\" (UniqueName: \"kubernetes.io/projected/361d0f4d-9529-461d-b89e-8e0cd8ca67ec-kube-api-access-qdc2b\") pod \"kube-state-metrics-84c885658c-7tlp8\" (UID: \"361d0f4d-9529-461d-b89e-8e0cd8ca67ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7tlp8"
Sep 17 17:58:46.860865 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.860869    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/361d0f4d-9529-461d-b89e-8e0cd8ca67ec-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-7tlp8\" (UID: \"361d0f4d-9529-461d-b89e-8e0cd8ca67ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7tlp8"
Sep 17 17:58:46.861069 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.860885    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8069ed7-c6a1-41cf-ba38-f21a714557e7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-2gv9q\" (UID: \"f8069ed7-c6a1-41cf-ba38-f21a714557e7\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2gv9q"
Sep 17 17:58:46.861069 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.860983    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/361d0f4d-9529-461d-b89e-8e0cd8ca67ec-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-7tlp8\" (UID: \"361d0f4d-9529-461d-b89e-8e0cd8ca67ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7tlp8"
Sep 17 17:58:46.861069 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.861014    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgx8b\" (UniqueName: \"kubernetes.io/projected/f8069ed7-c6a1-41cf-ba38-f21a714557e7-kube-api-access-sgx8b\") pod \"openshift-state-metrics-86c5bc79d6-2gv9q\" (UID: \"f8069ed7-c6a1-41cf-ba38-f21a714557e7\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2gv9q"
Sep 17 17:58:46.861069 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.861035    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/361d0f4d-9529-461d-b89e-8e0cd8ca67ec-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-7tlp8\" (UID: \"361d0f4d-9529-461d-b89e-8e0cd8ca67ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7tlp8"
Sep 17 17:58:46.861069 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.861059    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/361d0f4d-9529-461d-b89e-8e0cd8ca67ec-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-7tlp8\" (UID: \"361d0f4d-9529-461d-b89e-8e0cd8ca67ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7tlp8"
Sep 17 17:58:46.861225 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.861088    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8069ed7-c6a1-41cf-ba38-f21a714557e7-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-2gv9q\" (UID: \"f8069ed7-c6a1-41cf-ba38-f21a714557e7\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2gv9q"
Sep 17 17:58:46.861225 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.861113    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8069ed7-c6a1-41cf-ba38-f21a714557e7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-2gv9q\" (UID: \"f8069ed7-c6a1-41cf-ba38-f21a714557e7\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2gv9q"
Sep 17 17:58:46.861225 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.861131    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/361d0f4d-9529-461d-b89e-8e0cd8ca67ec-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-7tlp8\" (UID: \"361d0f4d-9529-461d-b89e-8e0cd8ca67ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7tlp8"
Sep 17 17:58:46.962194 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.962165    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/361d0f4d-9529-461d-b89e-8e0cd8ca67ec-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-7tlp8\" (UID: \"361d0f4d-9529-461d-b89e-8e0cd8ca67ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7tlp8"
Sep 17 17:58:46.962388 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.962202    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/442e5462-5952-46bc-819b-2aba220bea98-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5fr6x\" (UID: \"442e5462-5952-46bc-819b-2aba220bea98\") " pod="openshift-monitoring/node-exporter-5fr6x"
Sep 17 17:58:46.962388 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.962225    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgx8b\" (UniqueName: \"kubernetes.io/projected/f8069ed7-c6a1-41cf-ba38-f21a714557e7-kube-api-access-sgx8b\") pod \"openshift-state-metrics-86c5bc79d6-2gv9q\" (UID: \"f8069ed7-c6a1-41cf-ba38-f21a714557e7\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2gv9q"
Sep 17 17:58:46.962388 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.962356    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/442e5462-5952-46bc-819b-2aba220bea98-root\") pod \"node-exporter-5fr6x\" (UID: \"442e5462-5952-46bc-819b-2aba220bea98\") " pod="openshift-monitoring/node-exporter-5fr6x"
Sep 17 17:58:46.962563 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.962390    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/442e5462-5952-46bc-819b-2aba220bea98-node-exporter-tls\") pod \"node-exporter-5fr6x\" (UID: \"442e5462-5952-46bc-819b-2aba220bea98\") " pod="openshift-monitoring/node-exporter-5fr6x"
Sep 17 17:58:46.962563 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.962419    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/361d0f4d-9529-461d-b89e-8e0cd8ca67ec-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-7tlp8\" (UID: \"361d0f4d-9529-461d-b89e-8e0cd8ca67ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7tlp8"
Sep 17 17:58:46.962563 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.962456    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/361d0f4d-9529-461d-b89e-8e0cd8ca67ec-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-7tlp8\" (UID: \"361d0f4d-9529-461d-b89e-8e0cd8ca67ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7tlp8"
Sep 17 17:58:46.962563 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.962475    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/442e5462-5952-46bc-819b-2aba220bea98-sys\") pod \"node-exporter-5fr6x\" (UID: \"442e5462-5952-46bc-819b-2aba220bea98\") " pod="openshift-monitoring/node-exporter-5fr6x"
Sep 17 17:58:46.962563 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.962511    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8069ed7-c6a1-41cf-ba38-f21a714557e7-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-2gv9q\" (UID: \"f8069ed7-c6a1-41cf-ba38-f21a714557e7\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2gv9q"
Sep 17 17:58:46.962805 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.962562    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8069ed7-c6a1-41cf-ba38-f21a714557e7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-2gv9q\" (UID: \"f8069ed7-c6a1-41cf-ba38-f21a714557e7\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2gv9q"
Sep 17 17:58:46.962805 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.962617    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/361d0f4d-9529-461d-b89e-8e0cd8ca67ec-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-7tlp8\" (UID: \"361d0f4d-9529-461d-b89e-8e0cd8ca67ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7tlp8"
Sep 17 17:58:46.962805 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.962650    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/442e5462-5952-46bc-819b-2aba220bea98-node-exporter-textfile\") pod \"node-exporter-5fr6x\" (UID: \"442e5462-5952-46bc-819b-2aba220bea98\") " pod="openshift-monitoring/node-exporter-5fr6x"
Sep 17 17:58:46.962805 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.962689    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdc2b\" (UniqueName: \"kubernetes.io/projected/361d0f4d-9529-461d-b89e-8e0cd8ca67ec-kube-api-access-qdc2b\") pod \"kube-state-metrics-84c885658c-7tlp8\" (UID: \"361d0f4d-9529-461d-b89e-8e0cd8ca67ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7tlp8"
Sep 17 17:58:46.962805 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.962714    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-877qx\" (UniqueName: \"kubernetes.io/projected/442e5462-5952-46bc-819b-2aba220bea98-kube-api-access-877qx\") pod \"node-exporter-5fr6x\" (UID: \"442e5462-5952-46bc-819b-2aba220bea98\") " pod="openshift-monitoring/node-exporter-5fr6x"
Sep 17 17:58:46.962805 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.962739    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/442e5462-5952-46bc-819b-2aba220bea98-node-exporter-wtmp\") pod \"node-exporter-5fr6x\" (UID: \"442e5462-5952-46bc-819b-2aba220bea98\") " pod="openshift-monitoring/node-exporter-5fr6x"
Sep 17 17:58:46.962805 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.962773    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/442e5462-5952-46bc-819b-2aba220bea98-metrics-client-ca\") pod \"node-exporter-5fr6x\" (UID: \"442e5462-5952-46bc-819b-2aba220bea98\") " pod="openshift-monitoring/node-exporter-5fr6x"
Sep 17 17:58:46.962805 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.962802    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/361d0f4d-9529-461d-b89e-8e0cd8ca67ec-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-7tlp8\" (UID: \"361d0f4d-9529-461d-b89e-8e0cd8ca67ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7tlp8"
Sep 17 17:58:46.963203 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.962829    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8069ed7-c6a1-41cf-ba38-f21a714557e7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-2gv9q\" (UID: \"f8069ed7-c6a1-41cf-ba38-f21a714557e7\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2gv9q"
Sep 17 17:58:46.963203 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.962856    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/442e5462-5952-46bc-819b-2aba220bea98-node-exporter-accelerators-collector-config\") pod \"node-exporter-5fr6x\" (UID: \"442e5462-5952-46bc-819b-2aba220bea98\") " pod="openshift-monitoring/node-exporter-5fr6x"
Sep 17 17:58:46.963203 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.963053    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/361d0f4d-9529-461d-b89e-8e0cd8ca67ec-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-7tlp8\" (UID: \"361d0f4d-9529-461d-b89e-8e0cd8ca67ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7tlp8"
Sep 17 17:58:46.963339 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.963278    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8069ed7-c6a1-41cf-ba38-f21a714557e7-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-2gv9q\" (UID: \"f8069ed7-c6a1-41cf-ba38-f21a714557e7\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2gv9q"
Sep 17 17:58:46.963623 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.963601    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/361d0f4d-9529-461d-b89e-8e0cd8ca67ec-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-7tlp8\" (UID: \"361d0f4d-9529-461d-b89e-8e0cd8ca67ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7tlp8"
Sep 17 17:58:46.963732 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.963712    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/361d0f4d-9529-461d-b89e-8e0cd8ca67ec-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-7tlp8\" (UID: \"361d0f4d-9529-461d-b89e-8e0cd8ca67ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7tlp8"
Sep 17 17:58:46.965317 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.965290    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8069ed7-c6a1-41cf-ba38-f21a714557e7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-2gv9q\" (UID: \"f8069ed7-c6a1-41cf-ba38-f21a714557e7\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2gv9q"
Sep 17 17:58:46.965469 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.965444    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f8069ed7-c6a1-41cf-ba38-f21a714557e7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-2gv9q\" (UID: \"f8069ed7-c6a1-41cf-ba38-f21a714557e7\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2gv9q"
Sep 17 17:58:46.965591 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.965575    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/361d0f4d-9529-461d-b89e-8e0cd8ca67ec-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-7tlp8\" (UID: \"361d0f4d-9529-461d-b89e-8e0cd8ca67ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7tlp8"
Sep 17 17:58:46.965776 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.965756    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/361d0f4d-9529-461d-b89e-8e0cd8ca67ec-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-7tlp8\" (UID: \"361d0f4d-9529-461d-b89e-8e0cd8ca67ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7tlp8"
Sep 17 17:58:46.998559 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:46.998529    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdc2b\" (UniqueName: \"kubernetes.io/projected/361d0f4d-9529-461d-b89e-8e0cd8ca67ec-kube-api-access-qdc2b\") pod \"kube-state-metrics-84c885658c-7tlp8\" (UID: \"361d0f4d-9529-461d-b89e-8e0cd8ca67ec\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-7tlp8"
Sep 17 17:58:47.001236 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:47.001207    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgx8b\" (UniqueName: \"kubernetes.io/projected/f8069ed7-c6a1-41cf-ba38-f21a714557e7-kube-api-access-sgx8b\") pod \"openshift-state-metrics-86c5bc79d6-2gv9q\" (UID: \"f8069ed7-c6a1-41cf-ba38-f21a714557e7\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2gv9q"
Sep 17 17:58:47.004924 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:47.004887    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2gv9q"
Sep 17 17:58:47.049642 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:47.049608    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-7tlp8"
Sep 17 17:58:47.063987 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:47.063958    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/442e5462-5952-46bc-819b-2aba220bea98-root\") pod \"node-exporter-5fr6x\" (UID: \"442e5462-5952-46bc-819b-2aba220bea98\") " pod="openshift-monitoring/node-exporter-5fr6x"
Sep 17 17:58:47.064111 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:47.064004    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/442e5462-5952-46bc-819b-2aba220bea98-node-exporter-tls\") pod \"node-exporter-5fr6x\" (UID: \"442e5462-5952-46bc-819b-2aba220bea98\") " pod="openshift-monitoring/node-exporter-5fr6x"
Sep 17 17:58:47.064111 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:47.064051    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/442e5462-5952-46bc-819b-2aba220bea98-sys\") pod \"node-exporter-5fr6x\" (UID: \"442e5462-5952-46bc-819b-2aba220bea98\") " pod="openshift-monitoring/node-exporter-5fr6x"
Sep 17 17:58:47.064111 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:47.064071    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/442e5462-5952-46bc-819b-2aba220bea98-root\") pod \"node-exporter-5fr6x\" (UID: \"442e5462-5952-46bc-819b-2aba220bea98\") " pod="openshift-monitoring/node-exporter-5fr6x"
Sep 17 17:58:47.064111 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:47.064092    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/442e5462-5952-46bc-819b-2aba220bea98-node-exporter-textfile\") pod \"node-exporter-5fr6x\" (UID: \"442e5462-5952-46bc-819b-2aba220bea98\") " pod="openshift-monitoring/node-exporter-5fr6x"
Sep 17 17:58:47.064328 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:47.064138    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-877qx\" (UniqueName: \"kubernetes.io/projected/442e5462-5952-46bc-819b-2aba220bea98-kube-api-access-877qx\") pod \"node-exporter-5fr6x\" (UID: \"442e5462-5952-46bc-819b-2aba220bea98\") " pod="openshift-monitoring/node-exporter-5fr6x"
Sep 17 17:58:47.064328 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:47.064158    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/442e5462-5952-46bc-819b-2aba220bea98-node-exporter-wtmp\") pod \"node-exporter-5fr6x\" (UID: \"442e5462-5952-46bc-819b-2aba220bea98\") " pod="openshift-monitoring/node-exporter-5fr6x"
Sep 17 17:58:47.064328 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:47.064183    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/442e5462-5952-46bc-819b-2aba220bea98-metrics-client-ca\") pod \"node-exporter-5fr6x\" (UID: \"442e5462-5952-46bc-819b-2aba220bea98\") " pod="openshift-monitoring/node-exporter-5fr6x"
Sep 17 17:58:47.064328 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:47.064200    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/442e5462-5952-46bc-819b-2aba220bea98-node-exporter-accelerators-collector-config\") pod \"node-exporter-5fr6x\" (UID: \"442e5462-5952-46bc-819b-2aba220bea98\") " pod="openshift-monitoring/node-exporter-5fr6x"
Sep 17 17:58:47.064328 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:47.064236    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/442e5462-5952-46bc-819b-2aba220bea98-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5fr6x\" (UID: \"442e5462-5952-46bc-819b-2aba220bea98\") " pod="openshift-monitoring/node-exporter-5fr6x"
Sep 17 17:58:47.064571 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:47.064412    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/442e5462-5952-46bc-819b-2aba220bea98-node-exporter-textfile\") pod \"node-exporter-5fr6x\" (UID: \"442e5462-5952-46bc-819b-2aba220bea98\") " pod="openshift-monitoring/node-exporter-5fr6x"
Sep 17 17:58:47.064571 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:47.064435    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/442e5462-5952-46bc-819b-2aba220bea98-node-exporter-wtmp\") pod \"node-exporter-5fr6x\" (UID: \"442e5462-5952-46bc-819b-2aba220bea98\") " pod="openshift-monitoring/node-exporter-5fr6x"
Sep 17 17:58:47.064571 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:47.064479    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/442e5462-5952-46bc-819b-2aba220bea98-sys\") pod \"node-exporter-5fr6x\" (UID: \"442e5462-5952-46bc-819b-2aba220bea98\") " pod="openshift-monitoring/node-exporter-5fr6x"
Sep 17 17:58:47.065063 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:47.065042    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/442e5462-5952-46bc-819b-2aba220bea98-metrics-client-ca\") pod \"node-exporter-5fr6x\" (UID: \"442e5462-5952-46bc-819b-2aba220bea98\") " pod="openshift-monitoring/node-exporter-5fr6x"
Sep 17 17:58:47.065172 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:47.065127    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/442e5462-5952-46bc-819b-2aba220bea98-node-exporter-accelerators-collector-config\") pod \"node-exporter-5fr6x\" (UID: \"442e5462-5952-46bc-819b-2aba220bea98\") " pod="openshift-monitoring/node-exporter-5fr6x"
Sep 17 17:58:47.066617 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:47.066583    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/442e5462-5952-46bc-819b-2aba220bea98-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5fr6x\" (UID: \"442e5462-5952-46bc-819b-2aba220bea98\") " pod="openshift-monitoring/node-exporter-5fr6x"
Sep 17 17:58:47.067100 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:47.067071    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/442e5462-5952-46bc-819b-2aba220bea98-node-exporter-tls\") pod \"node-exporter-5fr6x\" (UID: \"442e5462-5952-46bc-819b-2aba220bea98\") " pod="openshift-monitoring/node-exporter-5fr6x"
Sep 17 17:58:47.085669 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:47.085639    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-877qx\" (UniqueName: \"kubernetes.io/projected/442e5462-5952-46bc-819b-2aba220bea98-kube-api-access-877qx\") pod \"node-exporter-5fr6x\" (UID: \"442e5462-5952-46bc-819b-2aba220bea98\") " pod="openshift-monitoring/node-exporter-5fr6x"
Sep 17 17:58:47.085804 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:47.085787    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5fr6x"
Sep 17 17:58:47.097249 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:58:47.097217    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod442e5462_5952_46bc_819b_2aba220bea98.slice/crio-2cb5ddab8fd1af28abfecc35398a0ac15e274e8499abf1533acea46c1cf984c5 WatchSource:0}: Error finding container 2cb5ddab8fd1af28abfecc35398a0ac15e274e8499abf1533acea46c1cf984c5: Status 404 returned error can't find the container with id 2cb5ddab8fd1af28abfecc35398a0ac15e274e8499abf1533acea46c1cf984c5
Sep 17 17:58:47.312433 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:47.312361    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-2gv9q"]
Sep 17 17:58:47.324099 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:58:47.324069    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8069ed7_c6a1_41cf_ba38_f21a714557e7.slice/crio-1ccbb6e60c5c53a374fc72100887c996e061ab349ab30b9cd2c047de28e04f1f WatchSource:0}: Error finding container 1ccbb6e60c5c53a374fc72100887c996e061ab349ab30b9cd2c047de28e04f1f: Status 404 returned error can't find the container with id 1ccbb6e60c5c53a374fc72100887c996e061ab349ab30b9cd2c047de28e04f1f
Sep 17 17:58:47.353032 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:47.353005    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-7tlp8"]
Sep 17 17:58:47.359852 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:58:47.359828    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod361d0f4d_9529_461d_b89e_8e0cd8ca67ec.slice/crio-8d7d3c93cb93df9082ceef6d80bd817fad7dbd887728a43871c27000e8a60c9d WatchSource:0}: Error finding container 8d7d3c93cb93df9082ceef6d80bd817fad7dbd887728a43871c27000e8a60c9d: Status 404 returned error can't find the container with id 8d7d3c93cb93df9082ceef6d80bd817fad7dbd887728a43871c27000e8a60c9d
Sep 17 17:58:47.872410 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:47.872374    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-7tlp8" event={"ID":"361d0f4d-9529-461d-b89e-8e0cd8ca67ec","Type":"ContainerStarted","Data":"8d7d3c93cb93df9082ceef6d80bd817fad7dbd887728a43871c27000e8a60c9d"}
Sep 17 17:58:47.874927 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:47.874879    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2gv9q" event={"ID":"f8069ed7-c6a1-41cf-ba38-f21a714557e7","Type":"ContainerStarted","Data":"96b49b57fa3aeff8ab5f2a2894839aa6044a200f07e3e6b3ffd48e0b0879c633"}
Sep 17 17:58:47.875073 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:47.874966    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2gv9q" event={"ID":"f8069ed7-c6a1-41cf-ba38-f21a714557e7","Type":"ContainerStarted","Data":"ca8bc7af9421fd2916528cc3c4048f4dec15eae4c9b3e16d1fa808a4a101555e"}
Sep 17 17:58:47.875073 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:47.874996    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2gv9q" event={"ID":"f8069ed7-c6a1-41cf-ba38-f21a714557e7","Type":"ContainerStarted","Data":"1ccbb6e60c5c53a374fc72100887c996e061ab349ab30b9cd2c047de28e04f1f"}
Sep 17 17:58:47.876426 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:47.876396    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5fr6x" event={"ID":"442e5462-5952-46bc-819b-2aba220bea98","Type":"ContainerStarted","Data":"2cb5ddab8fd1af28abfecc35398a0ac15e274e8499abf1533acea46c1cf984c5"}
Sep 17 17:58:48.042173 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.041452    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:58:48.044720 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.044697    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.050795 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.049986    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 17:58:48.050795 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.050261    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 17:58:48.050795 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.050653    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 17:58:48.050795 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.050669    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 17:58:48.051088 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.050934    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 17:58:48.051088 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.051067    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 17:58:48.051195 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.051139    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9www5\""
Sep 17 17:58:48.051396 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.051380    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 17:58:48.051616 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.051515    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 17:58:48.052781 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.052751    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 17:58:48.096958 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.096830    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:58:48.175224 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.175187    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.175224 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.175234    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k29hc\" (UniqueName: \"kubernetes.io/projected/04e326ad-b292-43f5-b0de-8dc05dfb794f-kube-api-access-k29hc\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.175459 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.175256    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04e326ad-b292-43f5-b0de-8dc05dfb794f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.175459 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.175314    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-web-config\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.175459 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.175362    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.175459 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.175392    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/04e326ad-b292-43f5-b0de-8dc05dfb794f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.175459 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.175426    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.175709 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.175467    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-config-volume\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.175709 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.175568    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/04e326ad-b292-43f5-b0de-8dc05dfb794f-config-out\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.175709 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.175601    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04e326ad-b292-43f5-b0de-8dc05dfb794f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.175709 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.175640    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.175709 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.175669    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/04e326ad-b292-43f5-b0de-8dc05dfb794f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.175709 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.175701    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.276382 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.276350    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.276382 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.276385    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/04e326ad-b292-43f5-b0de-8dc05dfb794f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.276638 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.276414    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.276638 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.276453    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.276638 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.276475    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k29hc\" (UniqueName: \"kubernetes.io/projected/04e326ad-b292-43f5-b0de-8dc05dfb794f-kube-api-access-k29hc\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.276638 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.276500    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04e326ad-b292-43f5-b0de-8dc05dfb794f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.276638 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.276516    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-web-config\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.276638 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.276541    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.276638 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.276568    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/04e326ad-b292-43f5-b0de-8dc05dfb794f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.276638 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.276604    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.277102 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.276634    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-config-volume\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.277102 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.276708    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/04e326ad-b292-43f5-b0de-8dc05dfb794f-config-out\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.277102 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.276733    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04e326ad-b292-43f5-b0de-8dc05dfb794f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.278029 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.278000    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04e326ad-b292-43f5-b0de-8dc05dfb794f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.278624 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.278294    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04e326ad-b292-43f5-b0de-8dc05dfb794f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.279794 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.279759    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/04e326ad-b292-43f5-b0de-8dc05dfb794f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.279895 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.279791    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/04e326ad-b292-43f5-b0de-8dc05dfb794f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.280337 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.280277    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.280480 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.280453    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.280480 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.280633    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/04e326ad-b292-43f5-b0de-8dc05dfb794f-config-out\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.280480 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.280672    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-web-config\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.281091 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.281048    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.282078 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.282051    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-config-volume\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.282283 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.282263    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.282425 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.282407    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.304316 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.304289    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k29hc\" (UniqueName: \"kubernetes.io/projected/04e326ad-b292-43f5-b0de-8dc05dfb794f-kube-api-access-k29hc\") pod \"alertmanager-main-0\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.357349 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.357320    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 17:58:48.882452 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.882418    2581 generic.go:356] "Generic (PLEG): container finished" podID="442e5462-5952-46bc-819b-2aba220bea98" containerID="f6d282a7b4903463041e82094fb19e474472a593c7e3b871f474a34d682f87cf" exitCode=0
Sep 17 17:58:48.882851 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.882507    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5fr6x" event={"ID":"442e5462-5952-46bc-819b-2aba220bea98","Type":"ContainerDied","Data":"f6d282a7b4903463041e82094fb19e474472a593c7e3b871f474a34d682f87cf"}
Sep 17 17:58:48.885200 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.884862    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-7tlp8" event={"ID":"361d0f4d-9529-461d-b89e-8e0cd8ca67ec","Type":"ContainerStarted","Data":"b7ab59710df62bbf37d0629febba137022d0bc0b98c5e6b1f6b7d408445d7849"}
Sep 17 17:58:48.888866 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.888834    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2gv9q" event={"ID":"f8069ed7-c6a1-41cf-ba38-f21a714557e7","Type":"ContainerStarted","Data":"c6353656193cc01fd791c59bbe49e257fc4acfbc016856b50c962120963177fa"}
Sep 17 17:58:48.981020 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:48.980754    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-2gv9q" podStartSLOduration=1.639115458 podStartE2EDuration="2.980729123s" totalImagesPullingTime="1.341613665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:48.979539982 +0000 UTC m=+117.070161633" watchObservedRunningTime="2026-09-17 17:58:48.980729123 +0000 UTC m=+117.071350775"
Sep 17 17:58:49.090119 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:49.090022    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 17:58:49.094686 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:58:49.094660    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e326ad_b292_43f5_b0de_8dc05dfb794f.slice/crio-f141e66d79153fcbf6fce26c78aade02c3e4cf8d9e42aac23ad87d1ddc97c61d WatchSource:0}: Error finding container f141e66d79153fcbf6fce26c78aade02c3e4cf8d9e42aac23ad87d1ddc97c61d: Status 404 returned error can't find the container with id f141e66d79153fcbf6fce26c78aade02c3e4cf8d9e42aac23ad87d1ddc97c61d
Sep 17 17:58:49.894587 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:49.894544    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"04e326ad-b292-43f5-b0de-8dc05dfb794f","Type":"ContainerStarted","Data":"f141e66d79153fcbf6fce26c78aade02c3e4cf8d9e42aac23ad87d1ddc97c61d"}
Sep 17 17:58:49.897039 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:49.897012    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5fr6x" event={"ID":"442e5462-5952-46bc-819b-2aba220bea98","Type":"ContainerStarted","Data":"3a11a907933ba5285fdc8e92afffe9b5b31037c0dfff1ca18b6d363fadac9f64"}
Sep 17 17:58:49.897190 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:49.897073    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5fr6x" event={"ID":"442e5462-5952-46bc-819b-2aba220bea98","Type":"ContainerStarted","Data":"8acc3775d65d84c1165e7570cf063d8178bba348b5adec000b56d9a637a4c6c7"}
Sep 17 17:58:49.899502 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:49.899420    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-7tlp8" event={"ID":"361d0f4d-9529-461d-b89e-8e0cd8ca67ec","Type":"ContainerStarted","Data":"d5d56e6a3f7ae1d73465844f039ae6fdd5bb0662608837cf74f2f7d500bdb03f"}
Sep 17 17:58:49.899502 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:49.899472    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-7tlp8" event={"ID":"361d0f4d-9529-461d-b89e-8e0cd8ca67ec","Type":"ContainerStarted","Data":"d2a49a5b0c22a870044cd69f33c07b845483cf1352ce9fadaed09115aabbabae"}
Sep 17 17:58:49.943855 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:49.943807    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5fr6x" podStartSLOduration=3.002692596 podStartE2EDuration="3.943790437s" totalImagesPullingTime="857.477506ms" totalInitContainerRuntime="83.620335ms" isStatefulPod=true podCreationTimestamp="2026-09-17 17:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:49.943659697 +0000 UTC m=+118.034281349" watchObservedRunningTime="2026-09-17 17:58:49.943790437 +0000 UTC m=+118.034412088"
Sep 17 17:58:50.001043 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:50.000978    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-7tlp8" podStartSLOduration=2.5818984 podStartE2EDuration="4.000954837s" totalImagesPullingTime="1.419056437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:49.999892331 +0000 UTC m=+118.090513980" watchObservedRunningTime="2026-09-17 17:58:50.000954837 +0000 UTC m=+118.091576498"
Sep 17 17:58:50.904464 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:50.904427    2581 generic.go:356] "Generic (PLEG): container finished" podID="04e326ad-b292-43f5-b0de-8dc05dfb794f" containerID="d015fbc97c06224ec28cb7a3f604576dd3eea33e3f50547af19cc8aa4ca87414" exitCode=0
Sep 17 17:58:50.904933 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:50.904508    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"04e326ad-b292-43f5-b0de-8dc05dfb794f","Type":"ContainerDied","Data":"d015fbc97c06224ec28cb7a3f604576dd3eea33e3f50547af19cc8aa4ca87414"}
Sep 17 17:58:51.029782 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.029750    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6d69786d4-cqbxm"]
Sep 17 17:58:51.032206 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.032191    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm"
Sep 17 17:58:51.044604 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.044582    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 17:58:51.045640 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.045625    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 17:58:51.046304 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.046284    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 17:58:51.046718 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.046702    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7gipkqukrc4rg\""
Sep 17 17:58:51.046811 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.046699    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-pr28t\""
Sep 17 17:58:51.046811 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.046760    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 17:58:51.046811 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.046760    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 17:58:51.057026 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.057002    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6d69786d4-cqbxm"]
Sep 17 17:58:51.203652 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.203625    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ab31f5c3-3c86-4de6-b021-e56291f4ce44-secret-grpc-tls\") pod \"thanos-querier-6d69786d4-cqbxm\" (UID: \"ab31f5c3-3c86-4de6-b021-e56291f4ce44\") " pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm"
Sep 17 17:58:51.203808 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.203666    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6zc4\" (UniqueName: \"kubernetes.io/projected/ab31f5c3-3c86-4de6-b021-e56291f4ce44-kube-api-access-h6zc4\") pod \"thanos-querier-6d69786d4-cqbxm\" (UID: \"ab31f5c3-3c86-4de6-b021-e56291f4ce44\") " pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm"
Sep 17 17:58:51.203808 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.203688    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ab31f5c3-3c86-4de6-b021-e56291f4ce44-secret-thanos-querier-tls\") pod \"thanos-querier-6d69786d4-cqbxm\" (UID: \"ab31f5c3-3c86-4de6-b021-e56291f4ce44\") " pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm"
Sep 17 17:58:51.203808 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.203748    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab31f5c3-3c86-4de6-b021-e56291f4ce44-metrics-client-ca\") pod \"thanos-querier-6d69786d4-cqbxm\" (UID: \"ab31f5c3-3c86-4de6-b021-e56291f4ce44\") " pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm"
Sep 17 17:58:51.203808 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.203795    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ab31f5c3-3c86-4de6-b021-e56291f4ce44-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d69786d4-cqbxm\" (UID: \"ab31f5c3-3c86-4de6-b021-e56291f4ce44\") " pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm"
Sep 17 17:58:51.204030 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.203862    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ab31f5c3-3c86-4de6-b021-e56291f4ce44-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d69786d4-cqbxm\" (UID: \"ab31f5c3-3c86-4de6-b021-e56291f4ce44\") " pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm"
Sep 17 17:58:51.204030 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.203887    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ab31f5c3-3c86-4de6-b021-e56291f4ce44-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d69786d4-cqbxm\" (UID: \"ab31f5c3-3c86-4de6-b021-e56291f4ce44\") " pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm"
Sep 17 17:58:51.204030 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.203953    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ab31f5c3-3c86-4de6-b021-e56291f4ce44-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d69786d4-cqbxm\" (UID: \"ab31f5c3-3c86-4de6-b021-e56291f4ce44\") " pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm"
Sep 17 17:58:51.305155 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.305120    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ab31f5c3-3c86-4de6-b021-e56291f4ce44-secret-grpc-tls\") pod \"thanos-querier-6d69786d4-cqbxm\" (UID: \"ab31f5c3-3c86-4de6-b021-e56291f4ce44\") " pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm"
Sep 17 17:58:51.305302 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.305225    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6zc4\" (UniqueName: \"kubernetes.io/projected/ab31f5c3-3c86-4de6-b021-e56291f4ce44-kube-api-access-h6zc4\") pod \"thanos-querier-6d69786d4-cqbxm\" (UID: \"ab31f5c3-3c86-4de6-b021-e56291f4ce44\") " pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm"
Sep 17 17:58:51.305302 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.305256    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ab31f5c3-3c86-4de6-b021-e56291f4ce44-secret-thanos-querier-tls\") pod \"thanos-querier-6d69786d4-cqbxm\" (UID: \"ab31f5c3-3c86-4de6-b021-e56291f4ce44\") " pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm"
Sep 17 17:58:51.305459 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.305437    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab31f5c3-3c86-4de6-b021-e56291f4ce44-metrics-client-ca\") pod \"thanos-querier-6d69786d4-cqbxm\" (UID: \"ab31f5c3-3c86-4de6-b021-e56291f4ce44\") " pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm"
Sep 17 17:58:51.305516 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.305485    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ab31f5c3-3c86-4de6-b021-e56291f4ce44-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d69786d4-cqbxm\" (UID: \"ab31f5c3-3c86-4de6-b021-e56291f4ce44\") " pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm"
Sep 17 17:58:51.305568 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.305539    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ab31f5c3-3c86-4de6-b021-e56291f4ce44-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d69786d4-cqbxm\" (UID: \"ab31f5c3-3c86-4de6-b021-e56291f4ce44\") " pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm"
Sep 17 17:58:51.305624 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.305566    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ab31f5c3-3c86-4de6-b021-e56291f4ce44-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d69786d4-cqbxm\" (UID: \"ab31f5c3-3c86-4de6-b021-e56291f4ce44\") " pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm"
Sep 17 17:58:51.305624 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.305599    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ab31f5c3-3c86-4de6-b021-e56291f4ce44-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d69786d4-cqbxm\" (UID: \"ab31f5c3-3c86-4de6-b021-e56291f4ce44\") " pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm"
Sep 17 17:58:51.306188 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.306161    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab31f5c3-3c86-4de6-b021-e56291f4ce44-metrics-client-ca\") pod \"thanos-querier-6d69786d4-cqbxm\" (UID: \"ab31f5c3-3c86-4de6-b021-e56291f4ce44\") " pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm"
Sep 17 17:58:51.308285 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.308242    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ab31f5c3-3c86-4de6-b021-e56291f4ce44-secret-thanos-querier-tls\") pod \"thanos-querier-6d69786d4-cqbxm\" (UID: \"ab31f5c3-3c86-4de6-b021-e56291f4ce44\") " pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm"
Sep 17 17:58:51.308371 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.308340    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ab31f5c3-3c86-4de6-b021-e56291f4ce44-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d69786d4-cqbxm\" (UID: \"ab31f5c3-3c86-4de6-b021-e56291f4ce44\") " pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm"
Sep 17 17:58:51.308431 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.308384    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ab31f5c3-3c86-4de6-b021-e56291f4ce44-secret-grpc-tls\") pod \"thanos-querier-6d69786d4-cqbxm\" (UID: \"ab31f5c3-3c86-4de6-b021-e56291f4ce44\") " pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm"
Sep 17 17:58:51.308619 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.308597    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ab31f5c3-3c86-4de6-b021-e56291f4ce44-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d69786d4-cqbxm\" (UID: \"ab31f5c3-3c86-4de6-b021-e56291f4ce44\") " pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm"
Sep 17 17:58:51.308690 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.308625    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ab31f5c3-3c86-4de6-b021-e56291f4ce44-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d69786d4-cqbxm\" (UID: \"ab31f5c3-3c86-4de6-b021-e56291f4ce44\") " pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm"
Sep 17 17:58:51.308784 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.308766    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ab31f5c3-3c86-4de6-b021-e56291f4ce44-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d69786d4-cqbxm\" (UID: \"ab31f5c3-3c86-4de6-b021-e56291f4ce44\") " pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm"
Sep 17 17:58:51.315388 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.315351    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6zc4\" (UniqueName: \"kubernetes.io/projected/ab31f5c3-3c86-4de6-b021-e56291f4ce44-kube-api-access-h6zc4\") pod \"thanos-querier-6d69786d4-cqbxm\" (UID: \"ab31f5c3-3c86-4de6-b021-e56291f4ce44\") " pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm"
Sep 17 17:58:51.341234 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.341212    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm"
Sep 17 17:58:51.622221 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.622148    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6bd7bb56fc-7q69l"]
Sep 17 17:58:51.625061 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.625037    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6bd7bb56fc-7q69l"
Sep 17 17:58:51.630279 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.630253    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 17 17:58:51.630437 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.630416    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 17:58:51.631549 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.631527    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 17 17:58:51.631666 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.631534    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 17 17:58:51.631666 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.631534    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fcnhf765p4d13\""
Sep 17 17:58:51.631666 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.631534    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-jvxwd\""
Sep 17 17:58:51.661982 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.661863    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6d69786d4-cqbxm"]
Sep 17 17:58:51.663757 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:58:51.663717    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab31f5c3_3c86_4de6_b021_e56291f4ce44.slice/crio-6c6ef33ef059713096753f984c56f9ab581f29c1c22a18b61d0ea1984345fe5d WatchSource:0}: Error finding container 6c6ef33ef059713096753f984c56f9ab581f29c1c22a18b61d0ea1984345fe5d: Status 404 returned error can't find the container with id 6c6ef33ef059713096753f984c56f9ab581f29c1c22a18b61d0ea1984345fe5d
Sep 17 17:58:51.706205 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.706176    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6bd7bb56fc-7q69l"]
Sep 17 17:58:51.709296 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.709273    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6c7b7223-42cd-4b34-ab15-7f9953d22478-metrics-server-audit-profiles\") pod \"metrics-server-6bd7bb56fc-7q69l\" (UID: \"6c7b7223-42cd-4b34-ab15-7f9953d22478\") " pod="openshift-monitoring/metrics-server-6bd7bb56fc-7q69l"
Sep 17 17:58:51.709410 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.709303    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c7b7223-42cd-4b34-ab15-7f9953d22478-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6bd7bb56fc-7q69l\" (UID: \"6c7b7223-42cd-4b34-ab15-7f9953d22478\") " pod="openshift-monitoring/metrics-server-6bd7bb56fc-7q69l"
Sep 17 17:58:51.709410 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.709336    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6c7b7223-42cd-4b34-ab15-7f9953d22478-secret-metrics-server-tls\") pod \"metrics-server-6bd7bb56fc-7q69l\" (UID: \"6c7b7223-42cd-4b34-ab15-7f9953d22478\") " pod="openshift-monitoring/metrics-server-6bd7bb56fc-7q69l"
Sep 17 17:58:51.709410 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.709377    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6c7b7223-42cd-4b34-ab15-7f9953d22478-audit-log\") pod \"metrics-server-6bd7bb56fc-7q69l\" (UID: \"6c7b7223-42cd-4b34-ab15-7f9953d22478\") " pod="openshift-monitoring/metrics-server-6bd7bb56fc-7q69l"
Sep 17 17:58:51.709509 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.709409    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6c7b7223-42cd-4b34-ab15-7f9953d22478-client-ca-bundle\") pod \"metrics-server-6bd7bb56fc-7q69l\" (UID: \"6c7b7223-42cd-4b34-ab15-7f9953d22478\") " pod="openshift-monitoring/metrics-server-6bd7bb56fc-7q69l"
Sep 17 17:58:51.709509 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.709468    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6c7b7223-42cd-4b34-ab15-7f9953d22478-secret-metrics-server-client-certs\") pod \"metrics-server-6bd7bb56fc-7q69l\" (UID: \"6c7b7223-42cd-4b34-ab15-7f9953d22478\") " pod="openshift-monitoring/metrics-server-6bd7bb56fc-7q69l"
Sep 17 17:58:51.709509 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.709488    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr8gh\" (UniqueName: \"kubernetes.io/projected/6c7b7223-42cd-4b34-ab15-7f9953d22478-kube-api-access-fr8gh\") pod \"metrics-server-6bd7bb56fc-7q69l\" (UID: \"6c7b7223-42cd-4b34-ab15-7f9953d22478\") " pod="openshift-monitoring/metrics-server-6bd7bb56fc-7q69l"
Sep 17 17:58:51.809974 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.809931    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6c7b7223-42cd-4b34-ab15-7f9953d22478-client-ca-bundle\") pod \"metrics-server-6bd7bb56fc-7q69l\" (UID: \"6c7b7223-42cd-4b34-ab15-7f9953d22478\") " pod="openshift-monitoring/metrics-server-6bd7bb56fc-7q69l"
Sep 17 17:58:51.810154 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.810008    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6c7b7223-42cd-4b34-ab15-7f9953d22478-secret-metrics-server-client-certs\") pod \"metrics-server-6bd7bb56fc-7q69l\" (UID: \"6c7b7223-42cd-4b34-ab15-7f9953d22478\") " pod="openshift-monitoring/metrics-server-6bd7bb56fc-7q69l"
Sep 17 17:58:51.810154 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.810030    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr8gh\" (UniqueName: \"kubernetes.io/projected/6c7b7223-42cd-4b34-ab15-7f9953d22478-kube-api-access-fr8gh\") pod \"metrics-server-6bd7bb56fc-7q69l\" (UID: \"6c7b7223-42cd-4b34-ab15-7f9953d22478\") " pod="openshift-monitoring/metrics-server-6bd7bb56fc-7q69l"
Sep 17 17:58:51.810154 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.810089    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6c7b7223-42cd-4b34-ab15-7f9953d22478-metrics-server-audit-profiles\") pod \"metrics-server-6bd7bb56fc-7q69l\" (UID: \"6c7b7223-42cd-4b34-ab15-7f9953d22478\") " pod="openshift-monitoring/metrics-server-6bd7bb56fc-7q69l"
Sep 17 17:58:51.810154 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.810122    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c7b7223-42cd-4b34-ab15-7f9953d22478-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6bd7bb56fc-7q69l\" (UID: \"6c7b7223-42cd-4b34-ab15-7f9953d22478\") " pod="openshift-monitoring/metrics-server-6bd7bb56fc-7q69l"
Sep 17 17:58:51.810366 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.810167    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6c7b7223-42cd-4b34-ab15-7f9953d22478-secret-metrics-server-tls\") pod \"metrics-server-6bd7bb56fc-7q69l\" (UID: \"6c7b7223-42cd-4b34-ab15-7f9953d22478\") " pod="openshift-monitoring/metrics-server-6bd7bb56fc-7q69l"
Sep 17 17:58:51.810366 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.810201    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6c7b7223-42cd-4b34-ab15-7f9953d22478-audit-log\") pod \"metrics-server-6bd7bb56fc-7q69l\" (UID: \"6c7b7223-42cd-4b34-ab15-7f9953d22478\") " pod="openshift-monitoring/metrics-server-6bd7bb56fc-7q69l"
Sep 17 17:58:51.810871 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.810811    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6c7b7223-42cd-4b34-ab15-7f9953d22478-audit-log\") pod \"metrics-server-6bd7bb56fc-7q69l\" (UID: \"6c7b7223-42cd-4b34-ab15-7f9953d22478\") " pod="openshift-monitoring/metrics-server-6bd7bb56fc-7q69l"
Sep 17 17:58:51.811139 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.811109    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c7b7223-42cd-4b34-ab15-7f9953d22478-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6bd7bb56fc-7q69l\" (UID: \"6c7b7223-42cd-4b34-ab15-7f9953d22478\") " pod="openshift-monitoring/metrics-server-6bd7bb56fc-7q69l"
Sep 17 17:58:51.811238 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.811211    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6c7b7223-42cd-4b34-ab15-7f9953d22478-metrics-server-audit-profiles\") pod \"metrics-server-6bd7bb56fc-7q69l\" (UID: \"6c7b7223-42cd-4b34-ab15-7f9953d22478\") " pod="openshift-monitoring/metrics-server-6bd7bb56fc-7q69l"
Sep 17 17:58:51.813101 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.813077    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6c7b7223-42cd-4b34-ab15-7f9953d22478-client-ca-bundle\") pod \"metrics-server-6bd7bb56fc-7q69l\" (UID: \"6c7b7223-42cd-4b34-ab15-7f9953d22478\") " pod="openshift-monitoring/metrics-server-6bd7bb56fc-7q69l"
Sep 17 17:58:51.813188 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.813131    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6c7b7223-42cd-4b34-ab15-7f9953d22478-secret-metrics-server-client-certs\") pod \"metrics-server-6bd7bb56fc-7q69l\" (UID: \"6c7b7223-42cd-4b34-ab15-7f9953d22478\") " pod="openshift-monitoring/metrics-server-6bd7bb56fc-7q69l"
Sep 17 17:58:51.813188 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.813157    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6c7b7223-42cd-4b34-ab15-7f9953d22478-secret-metrics-server-tls\") pod \"metrics-server-6bd7bb56fc-7q69l\" (UID: \"6c7b7223-42cd-4b34-ab15-7f9953d22478\") " pod="openshift-monitoring/metrics-server-6bd7bb56fc-7q69l"
Sep 17 17:58:51.823230 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.823210    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr8gh\" (UniqueName: \"kubernetes.io/projected/6c7b7223-42cd-4b34-ab15-7f9953d22478-kube-api-access-fr8gh\") pod \"metrics-server-6bd7bb56fc-7q69l\" (UID: \"6c7b7223-42cd-4b34-ab15-7f9953d22478\") " pod="openshift-monitoring/metrics-server-6bd7bb56fc-7q69l"
Sep 17 17:58:51.909424 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.909338    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm" event={"ID":"ab31f5c3-3c86-4de6-b021-e56291f4ce44","Type":"ContainerStarted","Data":"6c6ef33ef059713096753f984c56f9ab581f29c1c22a18b61d0ea1984345fe5d"}
Sep 17 17:58:51.936199 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:51.936163    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6bd7bb56fc-7q69l"
Sep 17 17:58:52.338024 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:52.337997    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6bd7bb56fc-7q69l"]
Sep 17 17:58:52.341939 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:58:52.341913    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c7b7223_42cd_4b34_ab15_7f9953d22478.slice/crio-03df6137ab7e0b7d6721b38b6772f8dc2b7fd6203681c933900dd248c441e43d WatchSource:0}: Error finding container 03df6137ab7e0b7d6721b38b6772f8dc2b7fd6203681c933900dd248c441e43d: Status 404 returned error can't find the container with id 03df6137ab7e0b7d6721b38b6772f8dc2b7fd6203681c933900dd248c441e43d
Sep 17 17:58:52.917191 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:52.917143    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6bd7bb56fc-7q69l" event={"ID":"6c7b7223-42cd-4b34-ab15-7f9953d22478","Type":"ContainerStarted","Data":"03df6137ab7e0b7d6721b38b6772f8dc2b7fd6203681c933900dd248c441e43d"}
Sep 17 17:58:52.921698 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:52.921666    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"04e326ad-b292-43f5-b0de-8dc05dfb794f","Type":"ContainerStarted","Data":"eda012fe779daaebe35ed32fc2b627f2e882b9d056e0cd21bddf78b9aec6cc95"}
Sep 17 17:58:52.921841 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:52.921745    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"04e326ad-b292-43f5-b0de-8dc05dfb794f","Type":"ContainerStarted","Data":"c45b1d750f58f8255c493db2574768fae82472348a668e1c9ddb9ae5c0ffe8fd"}
Sep 17 17:58:52.921841 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:52.921789    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"04e326ad-b292-43f5-b0de-8dc05dfb794f","Type":"ContainerStarted","Data":"5c5a2239f75c514759c29a9027c7672a2e4b289df34870b1b377739bafce2f4d"}
Sep 17 17:58:52.921841 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:52.921827    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"04e326ad-b292-43f5-b0de-8dc05dfb794f","Type":"ContainerStarted","Data":"78fbee8b10078b5ddb9b90f4848a368a4804fd5ed72802f2a048963f81ad3ee4"}
Sep 17 17:58:52.922046 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:52.921867    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"04e326ad-b292-43f5-b0de-8dc05dfb794f","Type":"ContainerStarted","Data":"a74ed5fc46ae02e7e21989ff9a9bafb637eff69b3af0dd0bc0157648b264aff7"}
Sep 17 17:58:54.932944 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:54.932892    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"04e326ad-b292-43f5-b0de-8dc05dfb794f","Type":"ContainerStarted","Data":"4e862a8a6fdca942c1af63a426a5f5539926198246ac31e639b680a2aad976f2"}
Sep 17 17:58:54.935802 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:54.935762    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm" event={"ID":"ab31f5c3-3c86-4de6-b021-e56291f4ce44","Type":"ContainerStarted","Data":"78eac4720e48725a9a918a4a405d5b0d5b48003eea24dd86c5ecd0d6ff48f1e4"}
Sep 17 17:58:54.935956 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:54.935836    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm" event={"ID":"ab31f5c3-3c86-4de6-b021-e56291f4ce44","Type":"ContainerStarted","Data":"117816060fc6fb41b21398aec892a7098ac264aa52bb482948cae43fd76ffdb9"}
Sep 17 17:58:54.935956 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:54.935878    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm" event={"ID":"ab31f5c3-3c86-4de6-b021-e56291f4ce44","Type":"ContainerStarted","Data":"f3061b3bf3ce3d1dd631c9461f1f4629f3f13c3ad1042f75b273c847b1968222"}
Sep 17 17:58:54.935956 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:54.935936    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm"
Sep 17 17:58:54.936115 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:54.935975    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm" event={"ID":"ab31f5c3-3c86-4de6-b021-e56291f4ce44","Type":"ContainerStarted","Data":"a2d4a865afc1d0a4760008e5daccda3b381f7001c3a9e74808636aeb6a9aff94"}
Sep 17 17:58:54.936115 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:54.936012    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm" event={"ID":"ab31f5c3-3c86-4de6-b021-e56291f4ce44","Type":"ContainerStarted","Data":"8eca143ae8f6fc29a52332faa4f6e9aaaae3b543db02c012a2d29b6d49f7bd33"}
Sep 17 17:58:54.936115 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:54.936049    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm" event={"ID":"ab31f5c3-3c86-4de6-b021-e56291f4ce44","Type":"ContainerStarted","Data":"27f177ccd8bb33f5ee7737e7a29e537daadb633e5119216dba1462bd3a05d673"}
Sep 17 17:58:54.937299 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:54.937279    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6bd7bb56fc-7q69l" event={"ID":"6c7b7223-42cd-4b34-ab15-7f9953d22478","Type":"ContainerStarted","Data":"8d16e558adfc961864ce4bd0e52c33356094b2bc420c96ebb8862cc091011565"}
Sep 17 17:58:54.975627 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:54.975576    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.118701485 podStartE2EDuration="7.975561678s" totalImagesPullingTime="3.839562063s" totalInitContainerRuntime="17.29813ms" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:54.974064554 +0000 UTC m=+123.064686208" watchObservedRunningTime="2026-09-17 17:58:54.975561678 +0000 UTC m=+123.066183327"
Sep 17 17:58:55.017181 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:55.017125    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm" podStartSLOduration=2.705949508 podStartE2EDuration="5.017107946s" totalImagesPullingTime="2.311158438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:55.016068625 +0000 UTC m=+123.106690275" watchObservedRunningTime="2026-09-17 17:58:55.017107946 +0000 UTC m=+123.107729596"
Sep 17 17:58:55.056008 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:55.055945    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6bd7bb56fc-7q69l" podStartSLOduration=2.4214630919999998 podStartE2EDuration="4.05592516s" totalImagesPullingTime="1.634462068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:55.054181397 +0000 UTC m=+123.144803039" watchObservedRunningTime="2026-09-17 17:58:55.05592516 +0000 UTC m=+123.146546811"
Sep 17 17:58:57.352847 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:57.352816    2581 scope.go:117] "RemoveContainer" containerID="25adf070ef000511fdf0bcfa6dbc92471aecf0754f82910c49da328a4e0689f8"
Sep 17 17:58:57.955251 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:57.955225    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/2.log"
Sep 17 17:58:57.955422 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:57.955317    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-l5vq7" event={"ID":"b41e1ab1-9be9-4db0-82da-f3dc6d863a24","Type":"ContainerStarted","Data":"2b9bfbe2c00cb551239ea99b7923fd7b9d3f9a8113145ae0e5be862fbf7ff3c2"}
Sep 17 17:58:57.955650 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:57.955603    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-l5vq7"
Sep 17 17:58:57.961199 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:57.961170    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-l5vq7"
Sep 17 17:58:58.025710 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:58:58.025648    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-l5vq7" podStartSLOduration=55.692334447 podStartE2EDuration="58.02562466s" totalImagesPullingTime="2.333290213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:58:57.981940757 +0000 UTC m=+126.072562404" watchObservedRunningTime="2026-09-17 17:58:58.02562466 +0000 UTC m=+126.116246312"
Sep 17 17:59:00.947542 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:00.947516    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6d69786d4-cqbxm"
Sep 17 17:59:02.192045 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:02.192008    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87bce58a-35d3-497c-8d82-7f925fe82b76-metrics-certs\") pod \"network-metrics-daemon-24pww\" (UID: \"87bce58a-35d3-497c-8d82-7f925fe82b76\") " pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:59:02.194310 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:02.194279    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87bce58a-35d3-497c-8d82-7f925fe82b76-metrics-certs\") pod \"network-metrics-daemon-24pww\" (UID: \"87bce58a-35d3-497c-8d82-7f925fe82b76\") " pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:59:02.265794 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:02.265759    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cf4sb\""
Sep 17 17:59:02.273581 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:02.273560    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-24pww"
Sep 17 17:59:02.547498 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:02.547470    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-24pww"]
Sep 17 17:59:02.550059 ip-10-0-130-28 kubenswrapper[2581]: W0917 17:59:02.550028    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87bce58a_35d3_497c_8d82_7f925fe82b76.slice/crio-c0b2ad2b9ed2055325b662f94fc78bd60369dc28f1fc9de4653fd2816314ccdb WatchSource:0}: Error finding container c0b2ad2b9ed2055325b662f94fc78bd60369dc28f1fc9de4653fd2816314ccdb: Status 404 returned error can't find the container with id c0b2ad2b9ed2055325b662f94fc78bd60369dc28f1fc9de4653fd2816314ccdb
Sep 17 17:59:02.977414 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:02.977373    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-24pww" event={"ID":"87bce58a-35d3-497c-8d82-7f925fe82b76","Type":"ContainerStarted","Data":"c0b2ad2b9ed2055325b662f94fc78bd60369dc28f1fc9de4653fd2816314ccdb"}
Sep 17 17:59:03.983146 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:03.983114    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-24pww" event={"ID":"87bce58a-35d3-497c-8d82-7f925fe82b76","Type":"ContainerStarted","Data":"dc6af204eb9f485c176b364363d544661b042d3beb1f7be6256f3d1b02d0e036"}
Sep 17 17:59:03.983546 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:03.983163    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-24pww" event={"ID":"87bce58a-35d3-497c-8d82-7f925fe82b76","Type":"ContainerStarted","Data":"253158dfba42ba5bcd44c367f1f5956207eaac9f7a6c3bd1aae89c4b2a77e278"}
Sep 17 17:59:04.027944 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:04.027872    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-24pww" podStartSLOduration=131.154329645 podStartE2EDuration="2m12.027855624s" totalImagesPullingTime="873.525979ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:56:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:59:04.025074928 +0000 UTC m=+132.115696578" watchObservedRunningTime="2026-09-17 17:59:04.027855624 +0000 UTC m=+132.118477278"
Sep 17 17:59:11.936303 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:11.936263    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6bd7bb56fc-7q69l"
Sep 17 17:59:11.936684 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:11.936330    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6bd7bb56fc-7q69l"
Sep 17 17:59:12.019222 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:12.019188    2581 generic.go:356] "Generic (PLEG): container finished" podID="08540561-3f4b-46ef-b859-d132f8488e4c" containerID="f499eb3b221b94fa2d48aa9d24dd178ce22ec7e110c6b9c84471205991eaa582" exitCode=0
Sep 17 17:59:12.019397 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:12.019262    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-lkct8" event={"ID":"08540561-3f4b-46ef-b859-d132f8488e4c","Type":"ContainerDied","Data":"f499eb3b221b94fa2d48aa9d24dd178ce22ec7e110c6b9c84471205991eaa582"}
Sep 17 17:59:12.019600 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:12.019580    2581 scope.go:117] "RemoveContainer" containerID="f499eb3b221b94fa2d48aa9d24dd178ce22ec7e110c6b9c84471205991eaa582"
Sep 17 17:59:13.027150 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:13.027107    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-lkct8" event={"ID":"08540561-3f4b-46ef-b859-d132f8488e4c","Type":"ContainerStarted","Data":"88ed4e88a452ca932d03218b95829c0699a690a7075c54069c39cf43f27ce6db"}
Sep 17 17:59:16.538511 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:16.538478    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_04e326ad-b292-43f5-b0de-8dc05dfb794f/init-config-reloader/0.log"
Sep 17 17:59:16.550203 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:16.550181    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_04e326ad-b292-43f5-b0de-8dc05dfb794f/alertmanager/0.log"
Sep 17 17:59:16.652376 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:16.652350    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_04e326ad-b292-43f5-b0de-8dc05dfb794f/config-reloader/0.log"
Sep 17 17:59:16.859012 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:16.858939    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_04e326ad-b292-43f5-b0de-8dc05dfb794f/kube-rbac-proxy-web/0.log"
Sep 17 17:59:17.050188 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:17.050159    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_04e326ad-b292-43f5-b0de-8dc05dfb794f/kube-rbac-proxy/0.log"
Sep 17 17:59:17.259714 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:17.259681    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_04e326ad-b292-43f5-b0de-8dc05dfb794f/kube-rbac-proxy-metric/0.log"
Sep 17 17:59:17.483320 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:17.483296    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_04e326ad-b292-43f5-b0de-8dc05dfb794f/prom-label-proxy/0.log"
Sep 17 17:59:17.656353 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:17.656260    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-zt5hc_9a99ade3-9dfb-4718-bf9b-07d713715bab/cluster-monitoring-operator/0.log"
Sep 17 17:59:17.847469 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:17.847435    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-7tlp8_361d0f4d-9529-461d-b89e-8e0cd8ca67ec/kube-state-metrics/0.log"
Sep 17 17:59:18.073116 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:18.073085    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-7tlp8_361d0f4d-9529-461d-b89e-8e0cd8ca67ec/kube-rbac-proxy-main/0.log"
Sep 17 17:59:18.245895 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:18.245869    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-7tlp8_361d0f4d-9529-461d-b89e-8e0cd8ca67ec/kube-rbac-proxy-self/0.log"
Sep 17 17:59:18.453256 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:18.453230    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6bd7bb56fc-7q69l_6c7b7223-42cd-4b34-ab15-7f9953d22478/metrics-server/0.log"
Sep 17 17:59:18.847545 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:18.847471    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5fr6x_442e5462-5952-46bc-819b-2aba220bea98/init-textfile/0.log"
Sep 17 17:59:19.049735 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:19.049704    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5fr6x_442e5462-5952-46bc-819b-2aba220bea98/node-exporter/0.log"
Sep 17 17:59:19.250837 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:19.250810    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5fr6x_442e5462-5952-46bc-819b-2aba220bea98/kube-rbac-proxy/0.log"
Sep 17 17:59:20.645684 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:20.645657    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-2gv9q_f8069ed7-c6a1-41cf-ba38-f21a714557e7/kube-rbac-proxy-main/0.log"
Sep 17 17:59:20.858516 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:20.858485    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-2gv9q_f8069ed7-c6a1-41cf-ba38-f21a714557e7/kube-rbac-proxy-self/0.log"
Sep 17 17:59:21.049402 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:21.049370    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-2gv9q_f8069ed7-c6a1-41cf-ba38-f21a714557e7/openshift-state-metrics/0.log"
Sep 17 17:59:23.247626 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:23.247594    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d69786d4-cqbxm_ab31f5c3-3c86-4de6-b021-e56291f4ce44/thanos-query/0.log"
Sep 17 17:59:23.449176 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:23.449149    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d69786d4-cqbxm_ab31f5c3-3c86-4de6-b021-e56291f4ce44/kube-rbac-proxy-web/0.log"
Sep 17 17:59:23.649088 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:23.649018    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d69786d4-cqbxm_ab31f5c3-3c86-4de6-b021-e56291f4ce44/kube-rbac-proxy/0.log"
Sep 17 17:59:23.845956 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:23.845920    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d69786d4-cqbxm_ab31f5c3-3c86-4de6-b021-e56291f4ce44/prom-label-proxy/0.log"
Sep 17 17:59:24.046763 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:24.046731    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d69786d4-cqbxm_ab31f5c3-3c86-4de6-b021-e56291f4ce44/kube-rbac-proxy-rules/0.log"
Sep 17 17:59:24.245512 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:24.245474    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d69786d4-cqbxm_ab31f5c3-3c86-4de6-b021-e56291f4ce44/kube-rbac-proxy-metrics/0.log"
Sep 17 17:59:24.645510 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:24.645485    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/2.log"
Sep 17 17:59:24.846429 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:24.846396    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/3.log"
Sep 17 17:59:31.942301 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:31.942274    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6bd7bb56fc-7q69l"
Sep 17 17:59:31.946057 ip-10-0-130-28 kubenswrapper[2581]: I0917 17:59:31.946036    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6bd7bb56fc-7q69l"
Sep 17 18:00:07.723750 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:07.723671    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:00:07.724263 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:07.723965    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="04e326ad-b292-43f5-b0de-8dc05dfb794f" containerName="alertmanager" containerID="cri-o://a74ed5fc46ae02e7e21989ff9a9bafb637eff69b3af0dd0bc0157648b264aff7" gracePeriod=120
Sep 17 18:00:07.724263 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:07.724053    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="04e326ad-b292-43f5-b0de-8dc05dfb794f" containerName="kube-rbac-proxy-web" containerID="cri-o://5c5a2239f75c514759c29a9027c7672a2e4b289df34870b1b377739bafce2f4d" gracePeriod=120
Sep 17 18:00:07.724263 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:07.724053    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="04e326ad-b292-43f5-b0de-8dc05dfb794f" containerName="kube-rbac-proxy-metric" containerID="cri-o://eda012fe779daaebe35ed32fc2b627f2e882b9d056e0cd21bddf78b9aec6cc95" gracePeriod=120
Sep 17 18:00:07.724263 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:07.724075    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="04e326ad-b292-43f5-b0de-8dc05dfb794f" containerName="prom-label-proxy" containerID="cri-o://4e862a8a6fdca942c1af63a426a5f5539926198246ac31e639b680a2aad976f2" gracePeriod=120
Sep 17 18:00:07.724263 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:07.724116    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="04e326ad-b292-43f5-b0de-8dc05dfb794f" containerName="config-reloader" containerID="cri-o://78fbee8b10078b5ddb9b90f4848a368a4804fd5ed72802f2a048963f81ad3ee4" gracePeriod=120
Sep 17 18:00:07.724263 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:07.724192    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="04e326ad-b292-43f5-b0de-8dc05dfb794f" containerName="kube-rbac-proxy" containerID="cri-o://c45b1d750f58f8255c493db2574768fae82472348a668e1c9ddb9ae5c0ffe8fd" gracePeriod=120
Sep 17 18:00:08.254510 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:08.254475    2581 generic.go:356] "Generic (PLEG): container finished" podID="04e326ad-b292-43f5-b0de-8dc05dfb794f" containerID="4e862a8a6fdca942c1af63a426a5f5539926198246ac31e639b680a2aad976f2" exitCode=0
Sep 17 18:00:08.254510 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:08.254503    2581 generic.go:356] "Generic (PLEG): container finished" podID="04e326ad-b292-43f5-b0de-8dc05dfb794f" containerID="c45b1d750f58f8255c493db2574768fae82472348a668e1c9ddb9ae5c0ffe8fd" exitCode=0
Sep 17 18:00:08.254510 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:08.254509    2581 generic.go:356] "Generic (PLEG): container finished" podID="04e326ad-b292-43f5-b0de-8dc05dfb794f" containerID="78fbee8b10078b5ddb9b90f4848a368a4804fd5ed72802f2a048963f81ad3ee4" exitCode=0
Sep 17 18:00:08.254510 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:08.254514    2581 generic.go:356] "Generic (PLEG): container finished" podID="04e326ad-b292-43f5-b0de-8dc05dfb794f" containerID="a74ed5fc46ae02e7e21989ff9a9bafb637eff69b3af0dd0bc0157648b264aff7" exitCode=0
Sep 17 18:00:08.254769 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:08.254540    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"04e326ad-b292-43f5-b0de-8dc05dfb794f","Type":"ContainerDied","Data":"4e862a8a6fdca942c1af63a426a5f5539926198246ac31e639b680a2aad976f2"}
Sep 17 18:00:08.254769 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:08.254604    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"04e326ad-b292-43f5-b0de-8dc05dfb794f","Type":"ContainerDied","Data":"c45b1d750f58f8255c493db2574768fae82472348a668e1c9ddb9ae5c0ffe8fd"}
Sep 17 18:00:08.254769 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:08.254633    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"04e326ad-b292-43f5-b0de-8dc05dfb794f","Type":"ContainerDied","Data":"78fbee8b10078b5ddb9b90f4848a368a4804fd5ed72802f2a048963f81ad3ee4"}
Sep 17 18:00:08.254769 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:08.254661    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"04e326ad-b292-43f5-b0de-8dc05dfb794f","Type":"ContainerDied","Data":"a74ed5fc46ae02e7e21989ff9a9bafb637eff69b3af0dd0bc0157648b264aff7"}
Sep 17 18:00:08.966929 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:08.966888    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.056546 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.056464    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-secret-alertmanager-kube-rbac-proxy\") pod \"04e326ad-b292-43f5-b0de-8dc05dfb794f\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") "
Sep 17 18:00:09.056546 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.056504    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"04e326ad-b292-43f5-b0de-8dc05dfb794f\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") "
Sep 17 18:00:09.056762 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.056549    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-cluster-tls-config\") pod \"04e326ad-b292-43f5-b0de-8dc05dfb794f\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") "
Sep 17 18:00:09.056762 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.056575    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-secret-alertmanager-main-tls\") pod \"04e326ad-b292-43f5-b0de-8dc05dfb794f\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") "
Sep 17 18:00:09.056762 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.056613    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-web-config\") pod \"04e326ad-b292-43f5-b0de-8dc05dfb794f\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") "
Sep 17 18:00:09.056762 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.056646    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/04e326ad-b292-43f5-b0de-8dc05dfb794f-tls-assets\") pod \"04e326ad-b292-43f5-b0de-8dc05dfb794f\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") "
Sep 17 18:00:09.056762 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.056676    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04e326ad-b292-43f5-b0de-8dc05dfb794f-alertmanager-trusted-ca-bundle\") pod \"04e326ad-b292-43f5-b0de-8dc05dfb794f\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") "
Sep 17 18:00:09.056762 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.056700    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-secret-alertmanager-kube-rbac-proxy-web\") pod \"04e326ad-b292-43f5-b0de-8dc05dfb794f\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") "
Sep 17 18:00:09.056762 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.056729    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04e326ad-b292-43f5-b0de-8dc05dfb794f-metrics-client-ca\") pod \"04e326ad-b292-43f5-b0de-8dc05dfb794f\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") "
Sep 17 18:00:09.057111 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.056777    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/04e326ad-b292-43f5-b0de-8dc05dfb794f-alertmanager-main-db\") pod \"04e326ad-b292-43f5-b0de-8dc05dfb794f\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") "
Sep 17 18:00:09.057111 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.056830    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k29hc\" (UniqueName: \"kubernetes.io/projected/04e326ad-b292-43f5-b0de-8dc05dfb794f-kube-api-access-k29hc\") pod \"04e326ad-b292-43f5-b0de-8dc05dfb794f\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") "
Sep 17 18:00:09.057111 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.056860    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/04e326ad-b292-43f5-b0de-8dc05dfb794f-config-out\") pod \"04e326ad-b292-43f5-b0de-8dc05dfb794f\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") "
Sep 17 18:00:09.057111 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.056897    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-config-volume\") pod \"04e326ad-b292-43f5-b0de-8dc05dfb794f\" (UID: \"04e326ad-b292-43f5-b0de-8dc05dfb794f\") "
Sep 17 18:00:09.057300 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.057268    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04e326ad-b292-43f5-b0de-8dc05dfb794f-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "04e326ad-b292-43f5-b0de-8dc05dfb794f" (UID: "04e326ad-b292-43f5-b0de-8dc05dfb794f"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:09.057954 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.057691    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04e326ad-b292-43f5-b0de-8dc05dfb794f-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "04e326ad-b292-43f5-b0de-8dc05dfb794f" (UID: "04e326ad-b292-43f5-b0de-8dc05dfb794f"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:09.058071 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.058034    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e326ad-b292-43f5-b0de-8dc05dfb794f-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "04e326ad-b292-43f5-b0de-8dc05dfb794f" (UID: "04e326ad-b292-43f5-b0de-8dc05dfb794f"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:09.060133 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.060077    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-config-volume" (OuterVolumeSpecName: "config-volume") pod "04e326ad-b292-43f5-b0de-8dc05dfb794f" (UID: "04e326ad-b292-43f5-b0de-8dc05dfb794f"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:09.060928 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.060866    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "04e326ad-b292-43f5-b0de-8dc05dfb794f" (UID: "04e326ad-b292-43f5-b0de-8dc05dfb794f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:09.061474 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.061438    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "04e326ad-b292-43f5-b0de-8dc05dfb794f" (UID: "04e326ad-b292-43f5-b0de-8dc05dfb794f"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:09.061586 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.061514    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "04e326ad-b292-43f5-b0de-8dc05dfb794f" (UID: "04e326ad-b292-43f5-b0de-8dc05dfb794f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:09.061690 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.061664    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e326ad-b292-43f5-b0de-8dc05dfb794f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "04e326ad-b292-43f5-b0de-8dc05dfb794f" (UID: "04e326ad-b292-43f5-b0de-8dc05dfb794f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:00:09.061893 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.061865    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e326ad-b292-43f5-b0de-8dc05dfb794f-kube-api-access-k29hc" (OuterVolumeSpecName: "kube-api-access-k29hc") pod "04e326ad-b292-43f5-b0de-8dc05dfb794f" (UID: "04e326ad-b292-43f5-b0de-8dc05dfb794f"). InnerVolumeSpecName "kube-api-access-k29hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:00:09.062010 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.061889    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "04e326ad-b292-43f5-b0de-8dc05dfb794f" (UID: "04e326ad-b292-43f5-b0de-8dc05dfb794f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:09.062447 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.062426    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e326ad-b292-43f5-b0de-8dc05dfb794f-config-out" (OuterVolumeSpecName: "config-out") pod "04e326ad-b292-43f5-b0de-8dc05dfb794f" (UID: "04e326ad-b292-43f5-b0de-8dc05dfb794f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:09.065345 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.065262    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "04e326ad-b292-43f5-b0de-8dc05dfb794f" (UID: "04e326ad-b292-43f5-b0de-8dc05dfb794f"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:09.071497 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.071467    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-web-config" (OuterVolumeSpecName: "web-config") pod "04e326ad-b292-43f5-b0de-8dc05dfb794f" (UID: "04e326ad-b292-43f5-b0de-8dc05dfb794f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:09.158152 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.158099    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k29hc\" (UniqueName: \"kubernetes.io/projected/04e326ad-b292-43f5-b0de-8dc05dfb794f-kube-api-access-k29hc\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:00:09.158152 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.158147    2581 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/04e326ad-b292-43f5-b0de-8dc05dfb794f-config-out\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:00:09.158152 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.158156    2581 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-config-volume\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:00:09.158152 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.158168    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:00:09.158423 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.158179    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:00:09.158423 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.158189    2581 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-cluster-tls-config\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:00:09.158423 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.158197    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-secret-alertmanager-main-tls\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:00:09.158423 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.158205    2581 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-web-config\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:00:09.158423 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.158213    2581 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/04e326ad-b292-43f5-b0de-8dc05dfb794f-tls-assets\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:00:09.158423 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.158221    2581 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04e326ad-b292-43f5-b0de-8dc05dfb794f-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:00:09.158423 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.158228    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/04e326ad-b292-43f5-b0de-8dc05dfb794f-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:00:09.158423 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.158239    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04e326ad-b292-43f5-b0de-8dc05dfb794f-metrics-client-ca\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:00:09.158423 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.158247    2581 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/04e326ad-b292-43f5-b0de-8dc05dfb794f-alertmanager-main-db\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:00:09.261412 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.261378    2581 generic.go:356] "Generic (PLEG): container finished" podID="04e326ad-b292-43f5-b0de-8dc05dfb794f" containerID="eda012fe779daaebe35ed32fc2b627f2e882b9d056e0cd21bddf78b9aec6cc95" exitCode=0
Sep 17 18:00:09.261412 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.261406    2581 generic.go:356] "Generic (PLEG): container finished" podID="04e326ad-b292-43f5-b0de-8dc05dfb794f" containerID="5c5a2239f75c514759c29a9027c7672a2e4b289df34870b1b377739bafce2f4d" exitCode=0
Sep 17 18:00:09.261619 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.261435    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"04e326ad-b292-43f5-b0de-8dc05dfb794f","Type":"ContainerDied","Data":"eda012fe779daaebe35ed32fc2b627f2e882b9d056e0cd21bddf78b9aec6cc95"}
Sep 17 18:00:09.261619 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.261448    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.261619 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.261491    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"04e326ad-b292-43f5-b0de-8dc05dfb794f","Type":"ContainerDied","Data":"5c5a2239f75c514759c29a9027c7672a2e4b289df34870b1b377739bafce2f4d"}
Sep 17 18:00:09.261619 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.261519    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"04e326ad-b292-43f5-b0de-8dc05dfb794f","Type":"ContainerDied","Data":"f141e66d79153fcbf6fce26c78aade02c3e4cf8d9e42aac23ad87d1ddc97c61d"}
Sep 17 18:00:09.261619 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.261555    2581 scope.go:117] "RemoveContainer" containerID="4e862a8a6fdca942c1af63a426a5f5539926198246ac31e639b680a2aad976f2"
Sep 17 18:00:09.269809 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.269787    2581 scope.go:117] "RemoveContainer" containerID="eda012fe779daaebe35ed32fc2b627f2e882b9d056e0cd21bddf78b9aec6cc95"
Sep 17 18:00:09.276837 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.276819    2581 scope.go:117] "RemoveContainer" containerID="c45b1d750f58f8255c493db2574768fae82472348a668e1c9ddb9ae5c0ffe8fd"
Sep 17 18:00:09.284010 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.283987    2581 scope.go:117] "RemoveContainer" containerID="5c5a2239f75c514759c29a9027c7672a2e4b289df34870b1b377739bafce2f4d"
Sep 17 18:00:09.287478 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.287454    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:00:09.291768 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.291752    2581 scope.go:117] "RemoveContainer" containerID="78fbee8b10078b5ddb9b90f4848a368a4804fd5ed72802f2a048963f81ad3ee4"
Sep 17 18:00:09.292184 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.292160    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:00:09.299229 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.299209    2581 scope.go:117] "RemoveContainer" containerID="a74ed5fc46ae02e7e21989ff9a9bafb637eff69b3af0dd0bc0157648b264aff7"
Sep 17 18:00:09.306731 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.306710    2581 scope.go:117] "RemoveContainer" containerID="d015fbc97c06224ec28cb7a3f604576dd3eea33e3f50547af19cc8aa4ca87414"
Sep 17 18:00:09.313686 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.313663    2581 scope.go:117] "RemoveContainer" containerID="4e862a8a6fdca942c1af63a426a5f5539926198246ac31e639b680a2aad976f2"
Sep 17 18:00:09.314063 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:00:09.314035    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e862a8a6fdca942c1af63a426a5f5539926198246ac31e639b680a2aad976f2\": container with ID starting with 4e862a8a6fdca942c1af63a426a5f5539926198246ac31e639b680a2aad976f2 not found: ID does not exist" containerID="4e862a8a6fdca942c1af63a426a5f5539926198246ac31e639b680a2aad976f2"
Sep 17 18:00:09.314150 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.314075    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e862a8a6fdca942c1af63a426a5f5539926198246ac31e639b680a2aad976f2"} err="failed to get container status \"4e862a8a6fdca942c1af63a426a5f5539926198246ac31e639b680a2aad976f2\": rpc error: code = NotFound desc = could not find container \"4e862a8a6fdca942c1af63a426a5f5539926198246ac31e639b680a2aad976f2\": container with ID starting with 4e862a8a6fdca942c1af63a426a5f5539926198246ac31e639b680a2aad976f2 not found: ID does not exist"
Sep 17 18:00:09.314150 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.314118    2581 scope.go:117] "RemoveContainer" containerID="eda012fe779daaebe35ed32fc2b627f2e882b9d056e0cd21bddf78b9aec6cc95"
Sep 17 18:00:09.314448 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:00:09.314420    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eda012fe779daaebe35ed32fc2b627f2e882b9d056e0cd21bddf78b9aec6cc95\": container with ID starting with eda012fe779daaebe35ed32fc2b627f2e882b9d056e0cd21bddf78b9aec6cc95 not found: ID does not exist" containerID="eda012fe779daaebe35ed32fc2b627f2e882b9d056e0cd21bddf78b9aec6cc95"
Sep 17 18:00:09.314607 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.314452    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eda012fe779daaebe35ed32fc2b627f2e882b9d056e0cd21bddf78b9aec6cc95"} err="failed to get container status \"eda012fe779daaebe35ed32fc2b627f2e882b9d056e0cd21bddf78b9aec6cc95\": rpc error: code = NotFound desc = could not find container \"eda012fe779daaebe35ed32fc2b627f2e882b9d056e0cd21bddf78b9aec6cc95\": container with ID starting with eda012fe779daaebe35ed32fc2b627f2e882b9d056e0cd21bddf78b9aec6cc95 not found: ID does not exist"
Sep 17 18:00:09.314607 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.314473    2581 scope.go:117] "RemoveContainer" containerID="c45b1d750f58f8255c493db2574768fae82472348a668e1c9ddb9ae5c0ffe8fd"
Sep 17 18:00:09.314767 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:00:09.314752    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c45b1d750f58f8255c493db2574768fae82472348a668e1c9ddb9ae5c0ffe8fd\": container with ID starting with c45b1d750f58f8255c493db2574768fae82472348a668e1c9ddb9ae5c0ffe8fd not found: ID does not exist" containerID="c45b1d750f58f8255c493db2574768fae82472348a668e1c9ddb9ae5c0ffe8fd"
Sep 17 18:00:09.314813 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.314772    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c45b1d750f58f8255c493db2574768fae82472348a668e1c9ddb9ae5c0ffe8fd"} err="failed to get container status \"c45b1d750f58f8255c493db2574768fae82472348a668e1c9ddb9ae5c0ffe8fd\": rpc error: code = NotFound desc = could not find container \"c45b1d750f58f8255c493db2574768fae82472348a668e1c9ddb9ae5c0ffe8fd\": container with ID starting with c45b1d750f58f8255c493db2574768fae82472348a668e1c9ddb9ae5c0ffe8fd not found: ID does not exist"
Sep 17 18:00:09.314813 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.314790    2581 scope.go:117] "RemoveContainer" containerID="5c5a2239f75c514759c29a9027c7672a2e4b289df34870b1b377739bafce2f4d"
Sep 17 18:00:09.315137 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:00:09.315121    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c5a2239f75c514759c29a9027c7672a2e4b289df34870b1b377739bafce2f4d\": container with ID starting with 5c5a2239f75c514759c29a9027c7672a2e4b289df34870b1b377739bafce2f4d not found: ID does not exist" containerID="5c5a2239f75c514759c29a9027c7672a2e4b289df34870b1b377739bafce2f4d"
Sep 17 18:00:09.315197 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.315140    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c5a2239f75c514759c29a9027c7672a2e4b289df34870b1b377739bafce2f4d"} err="failed to get container status \"5c5a2239f75c514759c29a9027c7672a2e4b289df34870b1b377739bafce2f4d\": rpc error: code = NotFound desc = could not find container \"5c5a2239f75c514759c29a9027c7672a2e4b289df34870b1b377739bafce2f4d\": container with ID starting with 5c5a2239f75c514759c29a9027c7672a2e4b289df34870b1b377739bafce2f4d not found: ID does not exist"
Sep 17 18:00:09.315197 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.315152    2581 scope.go:117] "RemoveContainer" containerID="78fbee8b10078b5ddb9b90f4848a368a4804fd5ed72802f2a048963f81ad3ee4"
Sep 17 18:00:09.315375 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:00:09.315355    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78fbee8b10078b5ddb9b90f4848a368a4804fd5ed72802f2a048963f81ad3ee4\": container with ID starting with 78fbee8b10078b5ddb9b90f4848a368a4804fd5ed72802f2a048963f81ad3ee4 not found: ID does not exist" containerID="78fbee8b10078b5ddb9b90f4848a368a4804fd5ed72802f2a048963f81ad3ee4"
Sep 17 18:00:09.315410 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.315379    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78fbee8b10078b5ddb9b90f4848a368a4804fd5ed72802f2a048963f81ad3ee4"} err="failed to get container status \"78fbee8b10078b5ddb9b90f4848a368a4804fd5ed72802f2a048963f81ad3ee4\": rpc error: code = NotFound desc = could not find container \"78fbee8b10078b5ddb9b90f4848a368a4804fd5ed72802f2a048963f81ad3ee4\": container with ID starting with 78fbee8b10078b5ddb9b90f4848a368a4804fd5ed72802f2a048963f81ad3ee4 not found: ID does not exist"
Sep 17 18:00:09.315410 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.315403    2581 scope.go:117] "RemoveContainer" containerID="a74ed5fc46ae02e7e21989ff9a9bafb637eff69b3af0dd0bc0157648b264aff7"
Sep 17 18:00:09.315646 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:00:09.315629    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a74ed5fc46ae02e7e21989ff9a9bafb637eff69b3af0dd0bc0157648b264aff7\": container with ID starting with a74ed5fc46ae02e7e21989ff9a9bafb637eff69b3af0dd0bc0157648b264aff7 not found: ID does not exist" containerID="a74ed5fc46ae02e7e21989ff9a9bafb637eff69b3af0dd0bc0157648b264aff7"
Sep 17 18:00:09.315696 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.315647    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a74ed5fc46ae02e7e21989ff9a9bafb637eff69b3af0dd0bc0157648b264aff7"} err="failed to get container status \"a74ed5fc46ae02e7e21989ff9a9bafb637eff69b3af0dd0bc0157648b264aff7\": rpc error: code = NotFound desc = could not find container \"a74ed5fc46ae02e7e21989ff9a9bafb637eff69b3af0dd0bc0157648b264aff7\": container with ID starting with a74ed5fc46ae02e7e21989ff9a9bafb637eff69b3af0dd0bc0157648b264aff7 not found: ID does not exist"
Sep 17 18:00:09.315696 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.315661    2581 scope.go:117] "RemoveContainer" containerID="d015fbc97c06224ec28cb7a3f604576dd3eea33e3f50547af19cc8aa4ca87414"
Sep 17 18:00:09.315879 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:00:09.315864    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d015fbc97c06224ec28cb7a3f604576dd3eea33e3f50547af19cc8aa4ca87414\": container with ID starting with d015fbc97c06224ec28cb7a3f604576dd3eea33e3f50547af19cc8aa4ca87414 not found: ID does not exist" containerID="d015fbc97c06224ec28cb7a3f604576dd3eea33e3f50547af19cc8aa4ca87414"
Sep 17 18:00:09.315970 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.315882    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d015fbc97c06224ec28cb7a3f604576dd3eea33e3f50547af19cc8aa4ca87414"} err="failed to get container status \"d015fbc97c06224ec28cb7a3f604576dd3eea33e3f50547af19cc8aa4ca87414\": rpc error: code = NotFound desc = could not find container \"d015fbc97c06224ec28cb7a3f604576dd3eea33e3f50547af19cc8aa4ca87414\": container with ID starting with d015fbc97c06224ec28cb7a3f604576dd3eea33e3f50547af19cc8aa4ca87414 not found: ID does not exist"
Sep 17 18:00:09.315970 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.315893    2581 scope.go:117] "RemoveContainer" containerID="4e862a8a6fdca942c1af63a426a5f5539926198246ac31e639b680a2aad976f2"
Sep 17 18:00:09.316142 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.316127    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e862a8a6fdca942c1af63a426a5f5539926198246ac31e639b680a2aad976f2"} err="failed to get container status \"4e862a8a6fdca942c1af63a426a5f5539926198246ac31e639b680a2aad976f2\": rpc error: code = NotFound desc = could not find container \"4e862a8a6fdca942c1af63a426a5f5539926198246ac31e639b680a2aad976f2\": container with ID starting with 4e862a8a6fdca942c1af63a426a5f5539926198246ac31e639b680a2aad976f2 not found: ID does not exist"
Sep 17 18:00:09.316187 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.316142    2581 scope.go:117] "RemoveContainer" containerID="eda012fe779daaebe35ed32fc2b627f2e882b9d056e0cd21bddf78b9aec6cc95"
Sep 17 18:00:09.316375 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.316357    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eda012fe779daaebe35ed32fc2b627f2e882b9d056e0cd21bddf78b9aec6cc95"} err="failed to get container status \"eda012fe779daaebe35ed32fc2b627f2e882b9d056e0cd21bddf78b9aec6cc95\": rpc error: code = NotFound desc = could not find container \"eda012fe779daaebe35ed32fc2b627f2e882b9d056e0cd21bddf78b9aec6cc95\": container with ID starting with eda012fe779daaebe35ed32fc2b627f2e882b9d056e0cd21bddf78b9aec6cc95 not found: ID does not exist"
Sep 17 18:00:09.316375 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.316375    2581 scope.go:117] "RemoveContainer" containerID="c45b1d750f58f8255c493db2574768fae82472348a668e1c9ddb9ae5c0ffe8fd"
Sep 17 18:00:09.316578 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.316563    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c45b1d750f58f8255c493db2574768fae82472348a668e1c9ddb9ae5c0ffe8fd"} err="failed to get container status \"c45b1d750f58f8255c493db2574768fae82472348a668e1c9ddb9ae5c0ffe8fd\": rpc error: code = NotFound desc = could not find container \"c45b1d750f58f8255c493db2574768fae82472348a668e1c9ddb9ae5c0ffe8fd\": container with ID starting with c45b1d750f58f8255c493db2574768fae82472348a668e1c9ddb9ae5c0ffe8fd not found: ID does not exist"
Sep 17 18:00:09.316578 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.316578    2581 scope.go:117] "RemoveContainer" containerID="5c5a2239f75c514759c29a9027c7672a2e4b289df34870b1b377739bafce2f4d"
Sep 17 18:00:09.316785 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.316767    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c5a2239f75c514759c29a9027c7672a2e4b289df34870b1b377739bafce2f4d"} err="failed to get container status \"5c5a2239f75c514759c29a9027c7672a2e4b289df34870b1b377739bafce2f4d\": rpc error: code = NotFound desc = could not find container \"5c5a2239f75c514759c29a9027c7672a2e4b289df34870b1b377739bafce2f4d\": container with ID starting with 5c5a2239f75c514759c29a9027c7672a2e4b289df34870b1b377739bafce2f4d not found: ID does not exist"
Sep 17 18:00:09.316785 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.316784    2581 scope.go:117] "RemoveContainer" containerID="78fbee8b10078b5ddb9b90f4848a368a4804fd5ed72802f2a048963f81ad3ee4"
Sep 17 18:00:09.317021 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.316997    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78fbee8b10078b5ddb9b90f4848a368a4804fd5ed72802f2a048963f81ad3ee4"} err="failed to get container status \"78fbee8b10078b5ddb9b90f4848a368a4804fd5ed72802f2a048963f81ad3ee4\": rpc error: code = NotFound desc = could not find container \"78fbee8b10078b5ddb9b90f4848a368a4804fd5ed72802f2a048963f81ad3ee4\": container with ID starting with 78fbee8b10078b5ddb9b90f4848a368a4804fd5ed72802f2a048963f81ad3ee4 not found: ID does not exist"
Sep 17 18:00:09.317089 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.317021    2581 scope.go:117] "RemoveContainer" containerID="a74ed5fc46ae02e7e21989ff9a9bafb637eff69b3af0dd0bc0157648b264aff7"
Sep 17 18:00:09.317239 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.317223    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a74ed5fc46ae02e7e21989ff9a9bafb637eff69b3af0dd0bc0157648b264aff7"} err="failed to get container status \"a74ed5fc46ae02e7e21989ff9a9bafb637eff69b3af0dd0bc0157648b264aff7\": rpc error: code = NotFound desc = could not find container \"a74ed5fc46ae02e7e21989ff9a9bafb637eff69b3af0dd0bc0157648b264aff7\": container with ID starting with a74ed5fc46ae02e7e21989ff9a9bafb637eff69b3af0dd0bc0157648b264aff7 not found: ID does not exist"
Sep 17 18:00:09.317283 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.317239    2581 scope.go:117] "RemoveContainer" containerID="d015fbc97c06224ec28cb7a3f604576dd3eea33e3f50547af19cc8aa4ca87414"
Sep 17 18:00:09.317439 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.317424    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d015fbc97c06224ec28cb7a3f604576dd3eea33e3f50547af19cc8aa4ca87414"} err="failed to get container status \"d015fbc97c06224ec28cb7a3f604576dd3eea33e3f50547af19cc8aa4ca87414\": rpc error: code = NotFound desc = could not find container \"d015fbc97c06224ec28cb7a3f604576dd3eea33e3f50547af19cc8aa4ca87414\": container with ID starting with d015fbc97c06224ec28cb7a3f604576dd3eea33e3f50547af19cc8aa4ca87414 not found: ID does not exist"
Sep 17 18:00:09.323680 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.323653    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:00:09.324019 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.324003    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04e326ad-b292-43f5-b0de-8dc05dfb794f" containerName="prom-label-proxy"
Sep 17 18:00:09.324100 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.324021    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="04e326ad-b292-43f5-b0de-8dc05dfb794f" containerName="prom-label-proxy"
Sep 17 18:00:09.324100 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.324035    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04e326ad-b292-43f5-b0de-8dc05dfb794f" containerName="kube-rbac-proxy-web"
Sep 17 18:00:09.324100 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.324044    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="04e326ad-b292-43f5-b0de-8dc05dfb794f" containerName="kube-rbac-proxy-web"
Sep 17 18:00:09.324100 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.324057    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04e326ad-b292-43f5-b0de-8dc05dfb794f" containerName="init-config-reloader"
Sep 17 18:00:09.324100 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.324066    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="04e326ad-b292-43f5-b0de-8dc05dfb794f" containerName="init-config-reloader"
Sep 17 18:00:09.324100 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.324078    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04e326ad-b292-43f5-b0de-8dc05dfb794f" containerName="kube-rbac-proxy-metric"
Sep 17 18:00:09.324100 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.324096    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="04e326ad-b292-43f5-b0de-8dc05dfb794f" containerName="kube-rbac-proxy-metric"
Sep 17 18:00:09.324448 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.324106    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04e326ad-b292-43f5-b0de-8dc05dfb794f" containerName="kube-rbac-proxy"
Sep 17 18:00:09.324448 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.324115    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="04e326ad-b292-43f5-b0de-8dc05dfb794f" containerName="kube-rbac-proxy"
Sep 17 18:00:09.324448 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.324129    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04e326ad-b292-43f5-b0de-8dc05dfb794f" containerName="alertmanager"
Sep 17 18:00:09.324448 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.324137    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="04e326ad-b292-43f5-b0de-8dc05dfb794f" containerName="alertmanager"
Sep 17 18:00:09.324448 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.324157    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04e326ad-b292-43f5-b0de-8dc05dfb794f" containerName="config-reloader"
Sep 17 18:00:09.324448 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.324165    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="04e326ad-b292-43f5-b0de-8dc05dfb794f" containerName="config-reloader"
Sep 17 18:00:09.324448 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.324246    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="04e326ad-b292-43f5-b0de-8dc05dfb794f" containerName="prom-label-proxy"
Sep 17 18:00:09.324448 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.324258    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="04e326ad-b292-43f5-b0de-8dc05dfb794f" containerName="kube-rbac-proxy-web"
Sep 17 18:00:09.324448 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.324269    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="04e326ad-b292-43f5-b0de-8dc05dfb794f" containerName="alertmanager"
Sep 17 18:00:09.324448 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.324280    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="04e326ad-b292-43f5-b0de-8dc05dfb794f" containerName="kube-rbac-proxy-metric"
Sep 17 18:00:09.324448 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.324290    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="04e326ad-b292-43f5-b0de-8dc05dfb794f" containerName="kube-rbac-proxy"
Sep 17 18:00:09.324448 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.324299    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="04e326ad-b292-43f5-b0de-8dc05dfb794f" containerName="config-reloader"
Sep 17 18:00:09.327715 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.327696    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.333456 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.333427    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 18:00:09.333591 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.333476    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 18:00:09.333591 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.333484    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 18:00:09.333591 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.333427    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 18:00:09.333591 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.333427    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 18:00:09.333591 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.333558    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9www5\""
Sep 17 18:00:09.333591 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.333433    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 18:00:09.333852 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.333478    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 18:00:09.333852 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.333428    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 18:00:09.337706 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.337683    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 18:00:09.345344 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.345319    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:00:09.460127 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.460078    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/61ef0b04-cf59-4219-99d1-a30a40477762-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.460127 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.460130    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/61ef0b04-cf59-4219-99d1-a30a40477762-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.460368 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.460147    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/61ef0b04-cf59-4219-99d1-a30a40477762-web-config\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.460368 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.460244    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/61ef0b04-cf59-4219-99d1-a30a40477762-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.460368 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.460289    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9ctj\" (UniqueName: \"kubernetes.io/projected/61ef0b04-cf59-4219-99d1-a30a40477762-kube-api-access-q9ctj\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.460368 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.460325    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/61ef0b04-cf59-4219-99d1-a30a40477762-config-out\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.460368 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.460363    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/61ef0b04-cf59-4219-99d1-a30a40477762-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.460624 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.460410    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/61ef0b04-cf59-4219-99d1-a30a40477762-config-volume\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.460624 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.460430    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61ef0b04-cf59-4219-99d1-a30a40477762-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.460624 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.460456    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/61ef0b04-cf59-4219-99d1-a30a40477762-tls-assets\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.460624 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.460481    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/61ef0b04-cf59-4219-99d1-a30a40477762-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.460624 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.460530    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61ef0b04-cf59-4219-99d1-a30a40477762-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.460624 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.460581    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/61ef0b04-cf59-4219-99d1-a30a40477762-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.561269 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.561176    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/61ef0b04-cf59-4219-99d1-a30a40477762-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.561269 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.561219    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9ctj\" (UniqueName: \"kubernetes.io/projected/61ef0b04-cf59-4219-99d1-a30a40477762-kube-api-access-q9ctj\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.561269 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.561236    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/61ef0b04-cf59-4219-99d1-a30a40477762-config-out\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.561514 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.561349    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/61ef0b04-cf59-4219-99d1-a30a40477762-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.561514 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.561387    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/61ef0b04-cf59-4219-99d1-a30a40477762-config-volume\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.561514 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.561405    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61ef0b04-cf59-4219-99d1-a30a40477762-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.561514 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.561427    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/61ef0b04-cf59-4219-99d1-a30a40477762-tls-assets\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.561726 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.561549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/61ef0b04-cf59-4219-99d1-a30a40477762-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.561726 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.561599    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61ef0b04-cf59-4219-99d1-a30a40477762-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.561726 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.561637    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/61ef0b04-cf59-4219-99d1-a30a40477762-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.561726 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.561682    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/61ef0b04-cf59-4219-99d1-a30a40477762-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.561726 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.561694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/61ef0b04-cf59-4219-99d1-a30a40477762-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.562005 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.561742    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/61ef0b04-cf59-4219-99d1-a30a40477762-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.562005 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.561766    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/61ef0b04-cf59-4219-99d1-a30a40477762-web-config\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.562301 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.562278    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61ef0b04-cf59-4219-99d1-a30a40477762-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.562388 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.562335    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61ef0b04-cf59-4219-99d1-a30a40477762-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.564475 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.564439    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/61ef0b04-cf59-4219-99d1-a30a40477762-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.564599 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.564479    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/61ef0b04-cf59-4219-99d1-a30a40477762-config-volume\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.564599 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.564536    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/61ef0b04-cf59-4219-99d1-a30a40477762-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.564716 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.564607    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/61ef0b04-cf59-4219-99d1-a30a40477762-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.564716 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.564617    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/61ef0b04-cf59-4219-99d1-a30a40477762-web-config\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.564804 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.564741    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/61ef0b04-cf59-4219-99d1-a30a40477762-config-out\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.564866 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.564847    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/61ef0b04-cf59-4219-99d1-a30a40477762-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.565215 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.565199    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/61ef0b04-cf59-4219-99d1-a30a40477762-tls-assets\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.566327 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.566311    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/61ef0b04-cf59-4219-99d1-a30a40477762-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.570253 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.570232    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9ctj\" (UniqueName: \"kubernetes.io/projected/61ef0b04-cf59-4219-99d1-a30a40477762-kube-api-access-q9ctj\") pod \"alertmanager-main-0\" (UID: \"61ef0b04-cf59-4219-99d1-a30a40477762\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.641362 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.641322    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 18:00:09.926663 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:09.926265    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 18:00:09.929021 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:00:09.928994    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ef0b04_cf59_4219_99d1_a30a40477762.slice/crio-5d882967b4c46ccecbb9da80a9a965b756bc504e7d6efd165ac5c31325561233 WatchSource:0}: Error finding container 5d882967b4c46ccecbb9da80a9a965b756bc504e7d6efd165ac5c31325561233: Status 404 returned error can't find the container with id 5d882967b4c46ccecbb9da80a9a965b756bc504e7d6efd165ac5c31325561233
Sep 17 18:00:10.267327 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:10.267294    2581 generic.go:356] "Generic (PLEG): container finished" podID="61ef0b04-cf59-4219-99d1-a30a40477762" containerID="981006be930b0205075c97bec2577ec62644f5bec9a9f87d5337d1b09d03d83f" exitCode=0
Sep 17 18:00:10.267773 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:10.267373    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"61ef0b04-cf59-4219-99d1-a30a40477762","Type":"ContainerDied","Data":"981006be930b0205075c97bec2577ec62644f5bec9a9f87d5337d1b09d03d83f"}
Sep 17 18:00:10.267773 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:10.267433    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"61ef0b04-cf59-4219-99d1-a30a40477762","Type":"ContainerStarted","Data":"5d882967b4c46ccecbb9da80a9a965b756bc504e7d6efd165ac5c31325561233"}
Sep 17 18:00:10.358388 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:10.358353    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04e326ad-b292-43f5-b0de-8dc05dfb794f" path="/var/lib/kubelet/pods/04e326ad-b292-43f5-b0de-8dc05dfb794f/volumes"
Sep 17 18:00:11.276425 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.276389    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"61ef0b04-cf59-4219-99d1-a30a40477762","Type":"ContainerStarted","Data":"9d467c5a3bf0540cd4f0ad920d4b4591d7168d85b582495751e5e82038bdb02d"}
Sep 17 18:00:11.276803 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.276486    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"61ef0b04-cf59-4219-99d1-a30a40477762","Type":"ContainerStarted","Data":"75996df3d692ad3a3acfeb8e43488a079b9a9f8873e1227f6de60658b594c365"}
Sep 17 18:00:11.276803 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.276560    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"61ef0b04-cf59-4219-99d1-a30a40477762","Type":"ContainerStarted","Data":"c18300c10092659d921c7a651ab32e5d88512410705b5df0b5c3c1d79f711645"}
Sep 17 18:00:11.276803 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.276594    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"61ef0b04-cf59-4219-99d1-a30a40477762","Type":"ContainerStarted","Data":"0ea5051bb8a8fd362974c5cfb7d1bc0e295865c010fce49bb86d861aaaa567f3"}
Sep 17 18:00:11.276803 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.276618    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"61ef0b04-cf59-4219-99d1-a30a40477762","Type":"ContainerStarted","Data":"3d2e6b185a9e29c0b72c1c98543ffd4fa34ef1bb2fe5888b4a824c2a5f839c4f"}
Sep 17 18:00:11.276803 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.276653    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"61ef0b04-cf59-4219-99d1-a30a40477762","Type":"ContainerStarted","Data":"cbd28d3ee4b772e92cd0ab2216f0bd9243db4c7d44e9b1a22485719f8d6a4cf5"}
Sep 17 18:00:11.310869 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.310814    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.29461575 podStartE2EDuration="2.310798621s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.182871ms" isStatefulPod=false podCreationTimestamp="2026-09-17 18:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:00:11.310528562 +0000 UTC m=+199.401150240" watchObservedRunningTime="2026-09-17 18:00:11.310798621 +0000 UTC m=+199.401420271"
Sep 17 18:00:11.771168 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.771134    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-598c547cf5-x5628"]
Sep 17 18:00:11.774511 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.774494    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-598c547cf5-x5628"
Sep 17 18:00:11.779589 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.779559    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 18:00:11.779589 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.779590    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 18:00:11.781147 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.781121    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-7zbgn\""
Sep 17 18:00:11.781430 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.781407    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 18:00:11.781531 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.781503    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 18:00:11.781584 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.781540    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 18:00:11.789263 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.789237    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 18:00:11.799239 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.799204    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-598c547cf5-x5628"]
Sep 17 18:00:11.882122 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.882088    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xl2p\" (UniqueName: \"kubernetes.io/projected/ca913e5c-28e7-4ef8-9e6a-34c90557c838-kube-api-access-6xl2p\") pod \"telemeter-client-598c547cf5-x5628\" (UID: \"ca913e5c-28e7-4ef8-9e6a-34c90557c838\") " pod="openshift-monitoring/telemeter-client-598c547cf5-x5628"
Sep 17 18:00:11.882310 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.882138    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca913e5c-28e7-4ef8-9e6a-34c90557c838-telemeter-trusted-ca-bundle\") pod \"telemeter-client-598c547cf5-x5628\" (UID: \"ca913e5c-28e7-4ef8-9e6a-34c90557c838\") " pod="openshift-monitoring/telemeter-client-598c547cf5-x5628"
Sep 17 18:00:11.882310 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.882162    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ca913e5c-28e7-4ef8-9e6a-34c90557c838-federate-client-tls\") pod \"telemeter-client-598c547cf5-x5628\" (UID: \"ca913e5c-28e7-4ef8-9e6a-34c90557c838\") " pod="openshift-monitoring/telemeter-client-598c547cf5-x5628"
Sep 17 18:00:11.882310 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.882200    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca913e5c-28e7-4ef8-9e6a-34c90557c838-serving-certs-ca-bundle\") pod \"telemeter-client-598c547cf5-x5628\" (UID: \"ca913e5c-28e7-4ef8-9e6a-34c90557c838\") " pod="openshift-monitoring/telemeter-client-598c547cf5-x5628"
Sep 17 18:00:11.882310 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.882226    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ca913e5c-28e7-4ef8-9e6a-34c90557c838-telemeter-client-tls\") pod \"telemeter-client-598c547cf5-x5628\" (UID: \"ca913e5c-28e7-4ef8-9e6a-34c90557c838\") " pod="openshift-monitoring/telemeter-client-598c547cf5-x5628"
Sep 17 18:00:11.882310 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.882247    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca913e5c-28e7-4ef8-9e6a-34c90557c838-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-598c547cf5-x5628\" (UID: \"ca913e5c-28e7-4ef8-9e6a-34c90557c838\") " pod="openshift-monitoring/telemeter-client-598c547cf5-x5628"
Sep 17 18:00:11.882310 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.882272    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ca913e5c-28e7-4ef8-9e6a-34c90557c838-secret-telemeter-client\") pod \"telemeter-client-598c547cf5-x5628\" (UID: \"ca913e5c-28e7-4ef8-9e6a-34c90557c838\") " pod="openshift-monitoring/telemeter-client-598c547cf5-x5628"
Sep 17 18:00:11.882310 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.882288    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca913e5c-28e7-4ef8-9e6a-34c90557c838-metrics-client-ca\") pod \"telemeter-client-598c547cf5-x5628\" (UID: \"ca913e5c-28e7-4ef8-9e6a-34c90557c838\") " pod="openshift-monitoring/telemeter-client-598c547cf5-x5628"
Sep 17 18:00:11.983578 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.983539    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xl2p\" (UniqueName: \"kubernetes.io/projected/ca913e5c-28e7-4ef8-9e6a-34c90557c838-kube-api-access-6xl2p\") pod \"telemeter-client-598c547cf5-x5628\" (UID: \"ca913e5c-28e7-4ef8-9e6a-34c90557c838\") " pod="openshift-monitoring/telemeter-client-598c547cf5-x5628"
Sep 17 18:00:11.983756 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.983599    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca913e5c-28e7-4ef8-9e6a-34c90557c838-telemeter-trusted-ca-bundle\") pod \"telemeter-client-598c547cf5-x5628\" (UID: \"ca913e5c-28e7-4ef8-9e6a-34c90557c838\") " pod="openshift-monitoring/telemeter-client-598c547cf5-x5628"
Sep 17 18:00:11.983756 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.983629    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ca913e5c-28e7-4ef8-9e6a-34c90557c838-federate-client-tls\") pod \"telemeter-client-598c547cf5-x5628\" (UID: \"ca913e5c-28e7-4ef8-9e6a-34c90557c838\") " pod="openshift-monitoring/telemeter-client-598c547cf5-x5628"
Sep 17 18:00:11.983756 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.983667    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca913e5c-28e7-4ef8-9e6a-34c90557c838-serving-certs-ca-bundle\") pod \"telemeter-client-598c547cf5-x5628\" (UID: \"ca913e5c-28e7-4ef8-9e6a-34c90557c838\") " pod="openshift-monitoring/telemeter-client-598c547cf5-x5628"
Sep 17 18:00:11.983756 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.983690    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ca913e5c-28e7-4ef8-9e6a-34c90557c838-telemeter-client-tls\") pod \"telemeter-client-598c547cf5-x5628\" (UID: \"ca913e5c-28e7-4ef8-9e6a-34c90557c838\") " pod="openshift-monitoring/telemeter-client-598c547cf5-x5628"
Sep 17 18:00:11.983756 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.983713    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca913e5c-28e7-4ef8-9e6a-34c90557c838-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-598c547cf5-x5628\" (UID: \"ca913e5c-28e7-4ef8-9e6a-34c90557c838\") " pod="openshift-monitoring/telemeter-client-598c547cf5-x5628"
Sep 17 18:00:11.983756 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.983740    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ca913e5c-28e7-4ef8-9e6a-34c90557c838-secret-telemeter-client\") pod \"telemeter-client-598c547cf5-x5628\" (UID: \"ca913e5c-28e7-4ef8-9e6a-34c90557c838\") " pod="openshift-monitoring/telemeter-client-598c547cf5-x5628"
Sep 17 18:00:11.984108 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.983766    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca913e5c-28e7-4ef8-9e6a-34c90557c838-metrics-client-ca\") pod \"telemeter-client-598c547cf5-x5628\" (UID: \"ca913e5c-28e7-4ef8-9e6a-34c90557c838\") " pod="openshift-monitoring/telemeter-client-598c547cf5-x5628"
Sep 17 18:00:11.984651 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.984622    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca913e5c-28e7-4ef8-9e6a-34c90557c838-metrics-client-ca\") pod \"telemeter-client-598c547cf5-x5628\" (UID: \"ca913e5c-28e7-4ef8-9e6a-34c90557c838\") " pod="openshift-monitoring/telemeter-client-598c547cf5-x5628"
Sep 17 18:00:11.984773 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.984669    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca913e5c-28e7-4ef8-9e6a-34c90557c838-serving-certs-ca-bundle\") pod \"telemeter-client-598c547cf5-x5628\" (UID: \"ca913e5c-28e7-4ef8-9e6a-34c90557c838\") " pod="openshift-monitoring/telemeter-client-598c547cf5-x5628"
Sep 17 18:00:11.984773 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.984753    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca913e5c-28e7-4ef8-9e6a-34c90557c838-telemeter-trusted-ca-bundle\") pod \"telemeter-client-598c547cf5-x5628\" (UID: \"ca913e5c-28e7-4ef8-9e6a-34c90557c838\") " pod="openshift-monitoring/telemeter-client-598c547cf5-x5628"
Sep 17 18:00:11.986521 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.986500    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ca913e5c-28e7-4ef8-9e6a-34c90557c838-telemeter-client-tls\") pod \"telemeter-client-598c547cf5-x5628\" (UID: \"ca913e5c-28e7-4ef8-9e6a-34c90557c838\") " pod="openshift-monitoring/telemeter-client-598c547cf5-x5628"
Sep 17 18:00:11.986625 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.986571    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca913e5c-28e7-4ef8-9e6a-34c90557c838-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-598c547cf5-x5628\" (UID: \"ca913e5c-28e7-4ef8-9e6a-34c90557c838\") " pod="openshift-monitoring/telemeter-client-598c547cf5-x5628"
Sep 17 18:00:11.986755 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.986738    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ca913e5c-28e7-4ef8-9e6a-34c90557c838-secret-telemeter-client\") pod \"telemeter-client-598c547cf5-x5628\" (UID: \"ca913e5c-28e7-4ef8-9e6a-34c90557c838\") " pod="openshift-monitoring/telemeter-client-598c547cf5-x5628"
Sep 17 18:00:11.986814 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:11.986766    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ca913e5c-28e7-4ef8-9e6a-34c90557c838-federate-client-tls\") pod \"telemeter-client-598c547cf5-x5628\" (UID: \"ca913e5c-28e7-4ef8-9e6a-34c90557c838\") " pod="openshift-monitoring/telemeter-client-598c547cf5-x5628"
Sep 17 18:00:12.001773 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:12.001739    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xl2p\" (UniqueName: \"kubernetes.io/projected/ca913e5c-28e7-4ef8-9e6a-34c90557c838-kube-api-access-6xl2p\") pod \"telemeter-client-598c547cf5-x5628\" (UID: \"ca913e5c-28e7-4ef8-9e6a-34c90557c838\") " pod="openshift-monitoring/telemeter-client-598c547cf5-x5628"
Sep 17 18:00:12.084558 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:12.084475    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-598c547cf5-x5628"
Sep 17 18:00:12.363102 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:12.363077    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-598c547cf5-x5628"]
Sep 17 18:00:12.364870 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:00:12.364838    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca913e5c_28e7_4ef8_9e6a_34c90557c838.slice/crio-e36b71d2d4d88b7cce5be68420b3642a2261dde18c2a2a03ed0c84c5d9f22977 WatchSource:0}: Error finding container e36b71d2d4d88b7cce5be68420b3642a2261dde18c2a2a03ed0c84c5d9f22977: Status 404 returned error can't find the container with id e36b71d2d4d88b7cce5be68420b3642a2261dde18c2a2a03ed0c84c5d9f22977
Sep 17 18:00:13.286071 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:13.286031    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-598c547cf5-x5628" event={"ID":"ca913e5c-28e7-4ef8-9e6a-34c90557c838","Type":"ContainerStarted","Data":"e36b71d2d4d88b7cce5be68420b3642a2261dde18c2a2a03ed0c84c5d9f22977"}
Sep 17 18:00:14.292532 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:14.292493    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-598c547cf5-x5628" event={"ID":"ca913e5c-28e7-4ef8-9e6a-34c90557c838","Type":"ContainerStarted","Data":"f6f2311d544e0e283abdaa68285ee9c456ac3e63eed4fb74aae14b3fd3334dd2"}
Sep 17 18:00:14.292924 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:14.292562    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-598c547cf5-x5628" event={"ID":"ca913e5c-28e7-4ef8-9e6a-34c90557c838","Type":"ContainerStarted","Data":"8cbfc5b5b0b790c9f5fbc19d08377a09dbf0012b7a37a8431c77560d7228cbe3"}
Sep 17 18:00:14.292924 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:14.292589    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-598c547cf5-x5628" event={"ID":"ca913e5c-28e7-4ef8-9e6a-34c90557c838","Type":"ContainerStarted","Data":"dcd052b590b2dc44a0d4e5cbb5f309f1e6e4d61cc2eff08d96893e5f45b3c62f"}
Sep 17 18:00:14.338717 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:00:14.338665    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-598c547cf5-x5628" podStartSLOduration=1.895320122 podStartE2EDuration="3.338646525s" totalImagesPullingTime="1.443326403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:00:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:00:14.336238844 +0000 UTC m=+202.426860494" watchObservedRunningTime="2026-09-17 18:00:14.338646525 +0000 UTC m=+202.429268174"
Sep 17 18:01:52.283831 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:01:52.283806    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/2.log"
Sep 17 18:01:52.288405 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:01:52.288379    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/2.log"
Sep 17 18:01:52.295034 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:01:52.295013    2581 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 18:02:23.849736 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:23.849695    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6459bdd986-b9j5d"]
Sep 17 18:02:23.852964 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:23.852937    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6459bdd986-b9j5d"
Sep 17 18:02:23.856676 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:23.856650    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 18:02:23.856676 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:23.856665    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 18:02:23.856856 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:23.856651    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 18:02:23.856856 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:23.856654    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 18:02:23.856856 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:23.856654    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-jf449\""
Sep 17 18:02:23.861451 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:23.861427    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6459bdd986-b9j5d"]
Sep 17 18:02:23.931012 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:23.930981    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/03bd9871-d44f-43a8-812c-5045db192d12-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6459bdd986-b9j5d\" (UID: \"03bd9871-d44f-43a8-812c-5045db192d12\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6459bdd986-b9j5d"
Sep 17 18:02:23.931162 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:23.931046    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zckl\" (UniqueName: \"kubernetes.io/projected/03bd9871-d44f-43a8-812c-5045db192d12-kube-api-access-8zckl\") pod \"managed-serviceaccount-addon-agent-6459bdd986-b9j5d\" (UID: \"03bd9871-d44f-43a8-812c-5045db192d12\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6459bdd986-b9j5d"
Sep 17 18:02:24.032251 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.032217    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/03bd9871-d44f-43a8-812c-5045db192d12-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6459bdd986-b9j5d\" (UID: \"03bd9871-d44f-43a8-812c-5045db192d12\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6459bdd986-b9j5d"
Sep 17 18:02:24.032413 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.032286    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zckl\" (UniqueName: \"kubernetes.io/projected/03bd9871-d44f-43a8-812c-5045db192d12-kube-api-access-8zckl\") pod \"managed-serviceaccount-addon-agent-6459bdd986-b9j5d\" (UID: \"03bd9871-d44f-43a8-812c-5045db192d12\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6459bdd986-b9j5d"
Sep 17 18:02:24.034672 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.034646    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/03bd9871-d44f-43a8-812c-5045db192d12-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6459bdd986-b9j5d\" (UID: \"03bd9871-d44f-43a8-812c-5045db192d12\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6459bdd986-b9j5d"
Sep 17 18:02:24.042448 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.042423    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zckl\" (UniqueName: \"kubernetes.io/projected/03bd9871-d44f-43a8-812c-5045db192d12-kube-api-access-8zckl\") pod \"managed-serviceaccount-addon-agent-6459bdd986-b9j5d\" (UID: \"03bd9871-d44f-43a8-812c-5045db192d12\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6459bdd986-b9j5d"
Sep 17 18:02:24.053001 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.052980    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655449f589-gdl5d"]
Sep 17 18:02:24.056231 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.056216    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655449f589-gdl5d"
Sep 17 18:02:24.058869 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.058847    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 18:02:24.059037 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.058882    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 18:02:24.059037 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.058913    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 18:02:24.059037 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.058942    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 18:02:24.065476 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.065457    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655449f589-gdl5d"]
Sep 17 18:02:24.132817 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.132747    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8f02081b-42f0-4bbf-8a59-b4b3ac85147c-ca\") pod \"cluster-proxy-proxy-agent-655449f589-gdl5d\" (UID: \"8f02081b-42f0-4bbf-8a59-b4b3ac85147c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655449f589-gdl5d"
Sep 17 18:02:24.132817 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.132774    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8f02081b-42f0-4bbf-8a59-b4b3ac85147c-hub\") pod \"cluster-proxy-proxy-agent-655449f589-gdl5d\" (UID: \"8f02081b-42f0-4bbf-8a59-b4b3ac85147c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655449f589-gdl5d"
Sep 17 18:02:24.132817 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.132817    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4lps\" (UniqueName: \"kubernetes.io/projected/8f02081b-42f0-4bbf-8a59-b4b3ac85147c-kube-api-access-c4lps\") pod \"cluster-proxy-proxy-agent-655449f589-gdl5d\" (UID: \"8f02081b-42f0-4bbf-8a59-b4b3ac85147c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655449f589-gdl5d"
Sep 17 18:02:24.133096 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.132837    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8f02081b-42f0-4bbf-8a59-b4b3ac85147c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-655449f589-gdl5d\" (UID: \"8f02081b-42f0-4bbf-8a59-b4b3ac85147c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655449f589-gdl5d"
Sep 17 18:02:24.133096 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.132897    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f02081b-42f0-4bbf-8a59-b4b3ac85147c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-655449f589-gdl5d\" (UID: \"8f02081b-42f0-4bbf-8a59-b4b3ac85147c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655449f589-gdl5d"
Sep 17 18:02:24.133096 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.132985    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8f02081b-42f0-4bbf-8a59-b4b3ac85147c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-655449f589-gdl5d\" (UID: \"8f02081b-42f0-4bbf-8a59-b4b3ac85147c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655449f589-gdl5d"
Sep 17 18:02:24.176747 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.176722    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6459bdd986-b9j5d"
Sep 17 18:02:24.233435 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.233398    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4lps\" (UniqueName: \"kubernetes.io/projected/8f02081b-42f0-4bbf-8a59-b4b3ac85147c-kube-api-access-c4lps\") pod \"cluster-proxy-proxy-agent-655449f589-gdl5d\" (UID: \"8f02081b-42f0-4bbf-8a59-b4b3ac85147c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655449f589-gdl5d"
Sep 17 18:02:24.233553 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.233454    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8f02081b-42f0-4bbf-8a59-b4b3ac85147c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-655449f589-gdl5d\" (UID: \"8f02081b-42f0-4bbf-8a59-b4b3ac85147c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655449f589-gdl5d"
Sep 17 18:02:24.233553 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.233533    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f02081b-42f0-4bbf-8a59-b4b3ac85147c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-655449f589-gdl5d\" (UID: \"8f02081b-42f0-4bbf-8a59-b4b3ac85147c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655449f589-gdl5d"
Sep 17 18:02:24.233680 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.233615    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8f02081b-42f0-4bbf-8a59-b4b3ac85147c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-655449f589-gdl5d\" (UID: \"8f02081b-42f0-4bbf-8a59-b4b3ac85147c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655449f589-gdl5d"
Sep 17 18:02:24.233680 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.233665    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8f02081b-42f0-4bbf-8a59-b4b3ac85147c-ca\") pod \"cluster-proxy-proxy-agent-655449f589-gdl5d\" (UID: \"8f02081b-42f0-4bbf-8a59-b4b3ac85147c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655449f589-gdl5d"
Sep 17 18:02:24.233772 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.233687    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8f02081b-42f0-4bbf-8a59-b4b3ac85147c-hub\") pod \"cluster-proxy-proxy-agent-655449f589-gdl5d\" (UID: \"8f02081b-42f0-4bbf-8a59-b4b3ac85147c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655449f589-gdl5d"
Sep 17 18:02:24.234701 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.234673    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f02081b-42f0-4bbf-8a59-b4b3ac85147c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-655449f589-gdl5d\" (UID: \"8f02081b-42f0-4bbf-8a59-b4b3ac85147c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655449f589-gdl5d"
Sep 17 18:02:24.239404 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.239353    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8f02081b-42f0-4bbf-8a59-b4b3ac85147c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-655449f589-gdl5d\" (UID: \"8f02081b-42f0-4bbf-8a59-b4b3ac85147c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655449f589-gdl5d"
Sep 17 18:02:24.239511 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.239461    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8f02081b-42f0-4bbf-8a59-b4b3ac85147c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-655449f589-gdl5d\" (UID: \"8f02081b-42f0-4bbf-8a59-b4b3ac85147c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655449f589-gdl5d"
Sep 17 18:02:24.239938 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.239893    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8f02081b-42f0-4bbf-8a59-b4b3ac85147c-hub\") pod \"cluster-proxy-proxy-agent-655449f589-gdl5d\" (UID: \"8f02081b-42f0-4bbf-8a59-b4b3ac85147c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655449f589-gdl5d"
Sep 17 18:02:24.240752 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.240726    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8f02081b-42f0-4bbf-8a59-b4b3ac85147c-ca\") pod \"cluster-proxy-proxy-agent-655449f589-gdl5d\" (UID: \"8f02081b-42f0-4bbf-8a59-b4b3ac85147c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655449f589-gdl5d"
Sep 17 18:02:24.243301 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.243279    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4lps\" (UniqueName: \"kubernetes.io/projected/8f02081b-42f0-4bbf-8a59-b4b3ac85147c-kube-api-access-c4lps\") pod \"cluster-proxy-proxy-agent-655449f589-gdl5d\" (UID: \"8f02081b-42f0-4bbf-8a59-b4b3ac85147c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655449f589-gdl5d"
Sep 17 18:02:24.365842 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.365813    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655449f589-gdl5d"
Sep 17 18:02:24.445288 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.445266    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6459bdd986-b9j5d"]
Sep 17 18:02:24.447202 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:02:24.447178    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03bd9871_d44f_43a8_812c_5045db192d12.slice/crio-dfd63125bb8546963ed189854109c83469b115136918e41ddd0691562eb316d7 WatchSource:0}: Error finding container dfd63125bb8546963ed189854109c83469b115136918e41ddd0691562eb316d7: Status 404 returned error can't find the container with id dfd63125bb8546963ed189854109c83469b115136918e41ddd0691562eb316d7
Sep 17 18:02:24.449069 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.449053    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:02:24.635699 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.635668    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655449f589-gdl5d"]
Sep 17 18:02:24.638072 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:02:24.638040    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f02081b_42f0_4bbf_8a59_b4b3ac85147c.slice/crio-1cdf56fa01d782c470436eec6abc19c81cd4d1ee965ff4435e1177d4768fcbb8 WatchSource:0}: Error finding container 1cdf56fa01d782c470436eec6abc19c81cd4d1ee965ff4435e1177d4768fcbb8: Status 404 returned error can't find the container with id 1cdf56fa01d782c470436eec6abc19c81cd4d1ee965ff4435e1177d4768fcbb8
Sep 17 18:02:24.804849 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.804811    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6459bdd986-b9j5d" event={"ID":"03bd9871-d44f-43a8-812c-5045db192d12","Type":"ContainerStarted","Data":"dfd63125bb8546963ed189854109c83469b115136918e41ddd0691562eb316d7"}
Sep 17 18:02:24.805837 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:24.805815    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655449f589-gdl5d" event={"ID":"8f02081b-42f0-4bbf-8a59-b4b3ac85147c","Type":"ContainerStarted","Data":"1cdf56fa01d782c470436eec6abc19c81cd4d1ee965ff4435e1177d4768fcbb8"}
Sep 17 18:02:27.823500 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:27.823465    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6459bdd986-b9j5d" event={"ID":"03bd9871-d44f-43a8-812c-5045db192d12","Type":"ContainerStarted","Data":"3005dca382aa248c0d452d6e6267257e99d44f67d9d43ca06ddbfd64aa388db1"}
Sep 17 18:02:27.841745 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:27.841612    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6459bdd986-b9j5d" podStartSLOduration=2.242054556 podStartE2EDuration="4.841591351s" totalImagesPullingTime="2.599536795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:02:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:02:27.839952296 +0000 UTC m=+335.930573946" watchObservedRunningTime="2026-09-17 18:02:27.841591351 +0000 UTC m=+335.932213001"
Sep 17 18:02:29.834370 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:29.834282    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655449f589-gdl5d" event={"ID":"8f02081b-42f0-4bbf-8a59-b4b3ac85147c","Type":"ContainerStarted","Data":"8eb078c4d451a08186ddf1c7f6023c4e77b1d613b470e2799691cc7ebdd31fb2"}
Sep 17 18:02:29.834370 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:29.834324    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655449f589-gdl5d" event={"ID":"8f02081b-42f0-4bbf-8a59-b4b3ac85147c","Type":"ContainerStarted","Data":"927585b38ae61ff60dd800eea1874a412d2b6575186d36b00b3f240c8e15fed5"}
Sep 17 18:02:29.834370 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:29.834336    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655449f589-gdl5d" event={"ID":"8f02081b-42f0-4bbf-8a59-b4b3ac85147c","Type":"ContainerStarted","Data":"9018bc1c1c281379f91c866fa419a2dbfeed63e13bb29ca50cda6702ebdb29d6"}
Sep 17 18:02:29.856697 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:02:29.856646    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-655449f589-gdl5d" podStartSLOduration=1.302957007 podStartE2EDuration="5.856630282s" totalImagesPullingTime="4.553673275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:02:29.854952043 +0000 UTC m=+337.945573694" watchObservedRunningTime="2026-09-17 18:02:29.856630282 +0000 UTC m=+337.947251922"
Sep 17 18:05:43.707500 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.707464    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"]
Sep 17 18:05:43.710504 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.710480    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:05:43.713319 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.713288    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 18:05:43.713427 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.713318    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 18:05:43.713489 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.713468    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-cv7sw\""
Sep 17 18:05:43.713629 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.713608    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 18:05:43.722287 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.722260    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8421b2a7-bdb3-4dbd-af93-943395f1007f-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw\" (UID: \"8421b2a7-bdb3-4dbd-af93-943395f1007f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:05:43.722389 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.722307    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8421b2a7-bdb3-4dbd-af93-943395f1007f-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw\" (UID: \"8421b2a7-bdb3-4dbd-af93-943395f1007f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:05:43.722389 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.722374    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8421b2a7-bdb3-4dbd-af93-943395f1007f-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw\" (UID: \"8421b2a7-bdb3-4dbd-af93-943395f1007f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:05:43.722464 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.722398    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggflg\" (UniqueName: \"kubernetes.io/projected/8421b2a7-bdb3-4dbd-af93-943395f1007f-kube-api-access-ggflg\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw\" (UID: \"8421b2a7-bdb3-4dbd-af93-943395f1007f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:05:43.722497 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.722451    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8421b2a7-bdb3-4dbd-af93-943395f1007f-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw\" (UID: \"8421b2a7-bdb3-4dbd-af93-943395f1007f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:05:43.722530 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.722496    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8421b2a7-bdb3-4dbd-af93-943395f1007f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw\" (UID: \"8421b2a7-bdb3-4dbd-af93-943395f1007f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:05:43.722563 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.722524    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8421b2a7-bdb3-4dbd-af93-943395f1007f-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw\" (UID: \"8421b2a7-bdb3-4dbd-af93-943395f1007f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:05:43.722563 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.722552    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8421b2a7-bdb3-4dbd-af93-943395f1007f-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw\" (UID: \"8421b2a7-bdb3-4dbd-af93-943395f1007f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:05:43.722628 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.722599    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8421b2a7-bdb3-4dbd-af93-943395f1007f-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw\" (UID: \"8421b2a7-bdb3-4dbd-af93-943395f1007f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:05:43.722949 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.722931    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"]
Sep 17 18:05:43.823138 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.823109    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8421b2a7-bdb3-4dbd-af93-943395f1007f-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw\" (UID: \"8421b2a7-bdb3-4dbd-af93-943395f1007f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:05:43.823307 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.823159    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8421b2a7-bdb3-4dbd-af93-943395f1007f-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw\" (UID: \"8421b2a7-bdb3-4dbd-af93-943395f1007f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:05:43.823307 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.823198    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8421b2a7-bdb3-4dbd-af93-943395f1007f-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw\" (UID: \"8421b2a7-bdb3-4dbd-af93-943395f1007f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:05:43.823307 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.823221    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggflg\" (UniqueName: \"kubernetes.io/projected/8421b2a7-bdb3-4dbd-af93-943395f1007f-kube-api-access-ggflg\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw\" (UID: \"8421b2a7-bdb3-4dbd-af93-943395f1007f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:05:43.823430 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.823331    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8421b2a7-bdb3-4dbd-af93-943395f1007f-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw\" (UID: \"8421b2a7-bdb3-4dbd-af93-943395f1007f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:05:43.823430 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.823377    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8421b2a7-bdb3-4dbd-af93-943395f1007f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw\" (UID: \"8421b2a7-bdb3-4dbd-af93-943395f1007f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:05:43.823430 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.823409    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8421b2a7-bdb3-4dbd-af93-943395f1007f-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw\" (UID: \"8421b2a7-bdb3-4dbd-af93-943395f1007f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:05:43.823570 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.823440    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8421b2a7-bdb3-4dbd-af93-943395f1007f-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw\" (UID: \"8421b2a7-bdb3-4dbd-af93-943395f1007f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:05:43.823570 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.823504    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8421b2a7-bdb3-4dbd-af93-943395f1007f-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw\" (UID: \"8421b2a7-bdb3-4dbd-af93-943395f1007f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:05:43.823570 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.823549    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8421b2a7-bdb3-4dbd-af93-943395f1007f-workload-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw\" (UID: \"8421b2a7-bdb3-4dbd-af93-943395f1007f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:05:43.823725 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.823577    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8421b2a7-bdb3-4dbd-af93-943395f1007f-credential-socket\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw\" (UID: \"8421b2a7-bdb3-4dbd-af93-943395f1007f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:05:43.823725 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.823651    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8421b2a7-bdb3-4dbd-af93-943395f1007f-workload-certs\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw\" (UID: \"8421b2a7-bdb3-4dbd-af93-943395f1007f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:05:43.823884 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.823829    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8421b2a7-bdb3-4dbd-af93-943395f1007f-istio-data\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw\" (UID: \"8421b2a7-bdb3-4dbd-af93-943395f1007f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:05:43.824211 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.824188    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8421b2a7-bdb3-4dbd-af93-943395f1007f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw\" (UID: \"8421b2a7-bdb3-4dbd-af93-943395f1007f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:05:43.825510 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.825485    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8421b2a7-bdb3-4dbd-af93-943395f1007f-istio-envoy\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw\" (UID: \"8421b2a7-bdb3-4dbd-af93-943395f1007f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:05:43.825750 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.825731    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8421b2a7-bdb3-4dbd-af93-943395f1007f-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw\" (UID: \"8421b2a7-bdb3-4dbd-af93-943395f1007f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:05:43.832782 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.832764    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggflg\" (UniqueName: \"kubernetes.io/projected/8421b2a7-bdb3-4dbd-af93-943395f1007f-kube-api-access-ggflg\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw\" (UID: \"8421b2a7-bdb3-4dbd-af93-943395f1007f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:05:43.833452 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:43.833434    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8421b2a7-bdb3-4dbd-af93-943395f1007f-istio-token\") pod \"openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw\" (UID: \"8421b2a7-bdb3-4dbd-af93-943395f1007f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:05:44.022519 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:44.022415    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:05:44.297076 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:44.297050    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"]
Sep 17 18:05:44.298983 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:05:44.298959    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8421b2a7_bdb3_4dbd_af93_943395f1007f.slice/crio-21b84ee4d74b1c37118ce9c09df6b66ae5d0f048f97925f6ddd24c072534fa3a WatchSource:0}: Error finding container 21b84ee4d74b1c37118ce9c09df6b66ae5d0f048f97925f6ddd24c072534fa3a: Status 404 returned error can't find the container with id 21b84ee4d74b1c37118ce9c09df6b66ae5d0f048f97925f6ddd24c072534fa3a
Sep 17 18:05:44.610099 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:44.610013    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw" event={"ID":"8421b2a7-bdb3-4dbd-af93-943395f1007f","Type":"ContainerStarted","Data":"21b84ee4d74b1c37118ce9c09df6b66ae5d0f048f97925f6ddd24c072534fa3a"}
Sep 17 18:05:47.060054 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:47.060012    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:05:47.060336 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:47.060090    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:05:47.060336 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:47.060124    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:05:47.625423 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:47.625385    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw" event={"ID":"8421b2a7-bdb3-4dbd-af93-943395f1007f","Type":"ContainerStarted","Data":"d34e47c35aea396e918f89ba89ee72b7412414f8105bfcd2854e29726a9a9fa0"}
Sep 17 18:05:47.665611 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:47.665561    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw" podStartSLOduration=1.906827404 podStartE2EDuration="4.665546007s" totalImagesPullingTime="2.758718603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:05:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:05:47.664614154 +0000 UTC m=+535.755235806" watchObservedRunningTime="2026-09-17 18:05:47.665546007 +0000 UTC m=+535.756167657"
Sep 17 18:05:48.022615 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:48.022579    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:05:48.027097 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:48.027072    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:05:48.630005 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:48.629978    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:05:48.630976 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:48.630953    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:05:52.542977 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:52.542940    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5654b77c54-7n5n5"]
Sep 17 18:05:52.546326 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:52.546310    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5654b77c54-7n5n5"
Sep 17 18:05:52.550659 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:52.550634    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 17 18:05:52.550659 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:52.550652    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:05:52.550854 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:52.550654    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 18:05:52.550854 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:52.550694    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 17 18:05:52.550854 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:52.550704    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 17 18:05:52.550854 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:52.550633    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-6t74h\""
Sep 17 18:05:52.559710 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:52.559682    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5654b77c54-7n5n5"]
Sep 17 18:05:52.610060 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:52.610023    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/71d2bec5-4ce3-4c5a-bc52-78cc8fb08f6c-metrics-cert\") pod \"lws-controller-manager-5654b77c54-7n5n5\" (UID: \"71d2bec5-4ce3-4c5a-bc52-78cc8fb08f6c\") " pod="openshift-lws-operator/lws-controller-manager-5654b77c54-7n5n5"
Sep 17 18:05:52.610231 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:52.610097    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/71d2bec5-4ce3-4c5a-bc52-78cc8fb08f6c-manager-config\") pod \"lws-controller-manager-5654b77c54-7n5n5\" (UID: \"71d2bec5-4ce3-4c5a-bc52-78cc8fb08f6c\") " pod="openshift-lws-operator/lws-controller-manager-5654b77c54-7n5n5"
Sep 17 18:05:52.610231 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:52.610142    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8dtl\" (UniqueName: \"kubernetes.io/projected/71d2bec5-4ce3-4c5a-bc52-78cc8fb08f6c-kube-api-access-f8dtl\") pod \"lws-controller-manager-5654b77c54-7n5n5\" (UID: \"71d2bec5-4ce3-4c5a-bc52-78cc8fb08f6c\") " pod="openshift-lws-operator/lws-controller-manager-5654b77c54-7n5n5"
Sep 17 18:05:52.610231 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:52.610172    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71d2bec5-4ce3-4c5a-bc52-78cc8fb08f6c-cert\") pod \"lws-controller-manager-5654b77c54-7n5n5\" (UID: \"71d2bec5-4ce3-4c5a-bc52-78cc8fb08f6c\") " pod="openshift-lws-operator/lws-controller-manager-5654b77c54-7n5n5"
Sep 17 18:05:52.710588 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:52.710556    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/71d2bec5-4ce3-4c5a-bc52-78cc8fb08f6c-manager-config\") pod \"lws-controller-manager-5654b77c54-7n5n5\" (UID: \"71d2bec5-4ce3-4c5a-bc52-78cc8fb08f6c\") " pod="openshift-lws-operator/lws-controller-manager-5654b77c54-7n5n5"
Sep 17 18:05:52.710764 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:52.710596    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8dtl\" (UniqueName: \"kubernetes.io/projected/71d2bec5-4ce3-4c5a-bc52-78cc8fb08f6c-kube-api-access-f8dtl\") pod \"lws-controller-manager-5654b77c54-7n5n5\" (UID: \"71d2bec5-4ce3-4c5a-bc52-78cc8fb08f6c\") " pod="openshift-lws-operator/lws-controller-manager-5654b77c54-7n5n5"
Sep 17 18:05:52.710764 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:52.710630    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71d2bec5-4ce3-4c5a-bc52-78cc8fb08f6c-cert\") pod \"lws-controller-manager-5654b77c54-7n5n5\" (UID: \"71d2bec5-4ce3-4c5a-bc52-78cc8fb08f6c\") " pod="openshift-lws-operator/lws-controller-manager-5654b77c54-7n5n5"
Sep 17 18:05:52.710764 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:52.710652    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/71d2bec5-4ce3-4c5a-bc52-78cc8fb08f6c-metrics-cert\") pod \"lws-controller-manager-5654b77c54-7n5n5\" (UID: \"71d2bec5-4ce3-4c5a-bc52-78cc8fb08f6c\") " pod="openshift-lws-operator/lws-controller-manager-5654b77c54-7n5n5"
Sep 17 18:05:52.711354 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:52.711333    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/71d2bec5-4ce3-4c5a-bc52-78cc8fb08f6c-manager-config\") pod \"lws-controller-manager-5654b77c54-7n5n5\" (UID: \"71d2bec5-4ce3-4c5a-bc52-78cc8fb08f6c\") " pod="openshift-lws-operator/lws-controller-manager-5654b77c54-7n5n5"
Sep 17 18:05:52.713224 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:52.713198    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71d2bec5-4ce3-4c5a-bc52-78cc8fb08f6c-cert\") pod \"lws-controller-manager-5654b77c54-7n5n5\" (UID: \"71d2bec5-4ce3-4c5a-bc52-78cc8fb08f6c\") " pod="openshift-lws-operator/lws-controller-manager-5654b77c54-7n5n5"
Sep 17 18:05:52.713340 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:52.713291    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/71d2bec5-4ce3-4c5a-bc52-78cc8fb08f6c-metrics-cert\") pod \"lws-controller-manager-5654b77c54-7n5n5\" (UID: \"71d2bec5-4ce3-4c5a-bc52-78cc8fb08f6c\") " pod="openshift-lws-operator/lws-controller-manager-5654b77c54-7n5n5"
Sep 17 18:05:52.721464 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:52.721440    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8dtl\" (UniqueName: \"kubernetes.io/projected/71d2bec5-4ce3-4c5a-bc52-78cc8fb08f6c-kube-api-access-f8dtl\") pod \"lws-controller-manager-5654b77c54-7n5n5\" (UID: \"71d2bec5-4ce3-4c5a-bc52-78cc8fb08f6c\") " pod="openshift-lws-operator/lws-controller-manager-5654b77c54-7n5n5"
Sep 17 18:05:52.856179 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:52.856069    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5654b77c54-7n5n5"
Sep 17 18:05:53.138051 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:53.138022    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5654b77c54-7n5n5"]
Sep 17 18:05:53.139853 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:05:53.139826    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d2bec5_4ce3_4c5a_bc52_78cc8fb08f6c.slice/crio-87833ec202bf783e419d4a6958b0dfc1926c7503aac8bdcdbd22aaa974bd5e12 WatchSource:0}: Error finding container 87833ec202bf783e419d4a6958b0dfc1926c7503aac8bdcdbd22aaa974bd5e12: Status 404 returned error can't find the container with id 87833ec202bf783e419d4a6958b0dfc1926c7503aac8bdcdbd22aaa974bd5e12
Sep 17 18:05:53.650933 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:53.650889    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5654b77c54-7n5n5" event={"ID":"71d2bec5-4ce3-4c5a-bc52-78cc8fb08f6c","Type":"ContainerStarted","Data":"87833ec202bf783e419d4a6958b0dfc1926c7503aac8bdcdbd22aaa974bd5e12"}
Sep 17 18:05:56.666526 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:56.666491    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5654b77c54-7n5n5" event={"ID":"71d2bec5-4ce3-4c5a-bc52-78cc8fb08f6c","Type":"ContainerStarted","Data":"f75c0f50ca142c83aa55db193d2e08a3d08c1083411f906b8effb03bd52315d8"}
Sep 17 18:05:56.666895 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:56.666616    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5654b77c54-7n5n5"
Sep 17 18:05:56.687218 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:05:56.687149    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5654b77c54-7n5n5" podStartSLOduration=1.977434643 podStartE2EDuration="4.687132451s" totalImagesPullingTime="2.709697808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:05:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:05:56.686385738 +0000 UTC m=+544.777007390" watchObservedRunningTime="2026-09-17 18:05:56.687132451 +0000 UTC m=+544.777754101"
Sep 17 18:06:07.673457 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:06:07.673423    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5654b77c54-7n5n5"
Sep 17 18:06:31.809552 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:06:31.809514    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-d5ft4"]
Sep 17 18:06:31.822414 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:06:31.822385    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d5ft4"
Sep 17 18:06:31.822580 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:06:31.822512    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-d5ft4"]
Sep 17 18:06:31.826507 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:06:31.826477    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 17 18:06:31.826654 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:06:31.826560    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 17 18:06:31.826721 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:06:31.826664    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 18:06:31.826808 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:06:31.826789    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 18:06:31.827654 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:06:31.827635    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rvg55\""
Sep 17 18:06:31.859228 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:06:31.859195    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg6m8\" (UniqueName: \"kubernetes.io/projected/e64ea2e8-c9d8-4c78-8649-ebd887d79399-kube-api-access-fg6m8\") pod \"kuadrant-console-plugin-fbdd4f445-d5ft4\" (UID: \"e64ea2e8-c9d8-4c78-8649-ebd887d79399\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d5ft4"
Sep 17 18:06:31.859400 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:06:31.859237    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e64ea2e8-c9d8-4c78-8649-ebd887d79399-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-d5ft4\" (UID: \"e64ea2e8-c9d8-4c78-8649-ebd887d79399\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d5ft4"
Sep 17 18:06:31.859400 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:06:31.859276    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e64ea2e8-c9d8-4c78-8649-ebd887d79399-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-d5ft4\" (UID: \"e64ea2e8-c9d8-4c78-8649-ebd887d79399\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d5ft4"
Sep 17 18:06:31.960709 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:06:31.960676    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e64ea2e8-c9d8-4c78-8649-ebd887d79399-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-d5ft4\" (UID: \"e64ea2e8-c9d8-4c78-8649-ebd887d79399\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d5ft4"
Sep 17 18:06:31.960939 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:06:31.960775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg6m8\" (UniqueName: \"kubernetes.io/projected/e64ea2e8-c9d8-4c78-8649-ebd887d79399-kube-api-access-fg6m8\") pod \"kuadrant-console-plugin-fbdd4f445-d5ft4\" (UID: \"e64ea2e8-c9d8-4c78-8649-ebd887d79399\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d5ft4"
Sep 17 18:06:31.960939 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:06:31.960813    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e64ea2e8-c9d8-4c78-8649-ebd887d79399-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-d5ft4\" (UID: \"e64ea2e8-c9d8-4c78-8649-ebd887d79399\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d5ft4"
Sep 17 18:06:31.961072 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:06:31.960960    2581 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 17 18:06:31.961072 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:06:31.961030    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e64ea2e8-c9d8-4c78-8649-ebd887d79399-plugin-serving-cert podName:e64ea2e8-c9d8-4c78-8649-ebd887d79399 nodeName:}" failed. No retries permitted until 2026-09-17 18:06:32.461007712 +0000 UTC m=+580.551629355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/e64ea2e8-c9d8-4c78-8649-ebd887d79399-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-d5ft4" (UID: "e64ea2e8-c9d8-4c78-8649-ebd887d79399") : secret "plugin-serving-cert" not found
Sep 17 18:06:31.961479 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:06:31.961456    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e64ea2e8-c9d8-4c78-8649-ebd887d79399-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-d5ft4\" (UID: \"e64ea2e8-c9d8-4c78-8649-ebd887d79399\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d5ft4"
Sep 17 18:06:31.971971 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:06:31.971943    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg6m8\" (UniqueName: \"kubernetes.io/projected/e64ea2e8-c9d8-4c78-8649-ebd887d79399-kube-api-access-fg6m8\") pod \"kuadrant-console-plugin-fbdd4f445-d5ft4\" (UID: \"e64ea2e8-c9d8-4c78-8649-ebd887d79399\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d5ft4"
Sep 17 18:06:32.465512 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:06:32.465481    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e64ea2e8-c9d8-4c78-8649-ebd887d79399-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-d5ft4\" (UID: \"e64ea2e8-c9d8-4c78-8649-ebd887d79399\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d5ft4"
Sep 17 18:06:32.467946 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:06:32.467925    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e64ea2e8-c9d8-4c78-8649-ebd887d79399-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-d5ft4\" (UID: \"e64ea2e8-c9d8-4c78-8649-ebd887d79399\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d5ft4"
Sep 17 18:06:32.732866 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:06:32.732774    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d5ft4"
Sep 17 18:06:33.011194 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:06:33.011162    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-d5ft4"]
Sep 17 18:06:33.014047 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:06:33.014020    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode64ea2e8_c9d8_4c78_8649_ebd887d79399.slice/crio-49437ffe29e8154f1e1154fd879af3c2b807ab8cd7950c2a9f70b1faedb0dfd9 WatchSource:0}: Error finding container 49437ffe29e8154f1e1154fd879af3c2b807ab8cd7950c2a9f70b1faedb0dfd9: Status 404 returned error can't find the container with id 49437ffe29e8154f1e1154fd879af3c2b807ab8cd7950c2a9f70b1faedb0dfd9
Sep 17 18:06:33.819445 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:06:33.819406    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d5ft4" event={"ID":"e64ea2e8-c9d8-4c78-8649-ebd887d79399","Type":"ContainerStarted","Data":"49437ffe29e8154f1e1154fd879af3c2b807ab8cd7950c2a9f70b1faedb0dfd9"}
Sep 17 18:06:38.843657 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:06:38.843620    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d5ft4" event={"ID":"e64ea2e8-c9d8-4c78-8649-ebd887d79399","Type":"ContainerStarted","Data":"c485c721732d0e29cdeb241b5fbd9df8380e518c33598e2d91bb0f46188f2083"}
Sep 17 18:06:38.879701 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:06:38.879645    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-d5ft4" podStartSLOduration=2.727150975 podStartE2EDuration="7.879630043s" totalImagesPullingTime="5.152479068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:06:38.87821599 +0000 UTC m=+586.968837663" watchObservedRunningTime="2026-09-17 18:06:38.879630043 +0000 UTC m=+586.970251692"
Sep 17 18:06:52.317737 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:06:52.317710    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/2.log"
Sep 17 18:06:52.318139 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:06:52.317747    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/2.log"
Sep 17 18:07:17.434070 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:17.433992    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-sn8fz"]
Sep 17 18:07:17.437170 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:17.437153    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-sn8fz"
Sep 17 18:07:17.440033 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:17.440005    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 17 18:07:17.447285 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:17.447259    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-sn8fz"]
Sep 17 18:07:17.464549 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:17.464513    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/83cd949f-1579-45be-bbd8-728d45112ef3-config-file\") pod \"limitador-limitador-8588749977-sn8fz\" (UID: \"83cd949f-1579-45be-bbd8-728d45112ef3\") " pod="kuadrant-system/limitador-limitador-8588749977-sn8fz"
Sep 17 18:07:17.464724 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:17.464563    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdnjj\" (UniqueName: \"kubernetes.io/projected/83cd949f-1579-45be-bbd8-728d45112ef3-kube-api-access-jdnjj\") pod \"limitador-limitador-8588749977-sn8fz\" (UID: \"83cd949f-1579-45be-bbd8-728d45112ef3\") " pod="kuadrant-system/limitador-limitador-8588749977-sn8fz"
Sep 17 18:07:17.520359 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:17.520330    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-sn8fz"]
Sep 17 18:07:17.565103 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:17.565060    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/83cd949f-1579-45be-bbd8-728d45112ef3-config-file\") pod \"limitador-limitador-8588749977-sn8fz\" (UID: \"83cd949f-1579-45be-bbd8-728d45112ef3\") " pod="kuadrant-system/limitador-limitador-8588749977-sn8fz"
Sep 17 18:07:17.565263 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:17.565118    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdnjj\" (UniqueName: \"kubernetes.io/projected/83cd949f-1579-45be-bbd8-728d45112ef3-kube-api-access-jdnjj\") pod \"limitador-limitador-8588749977-sn8fz\" (UID: \"83cd949f-1579-45be-bbd8-728d45112ef3\") " pod="kuadrant-system/limitador-limitador-8588749977-sn8fz"
Sep 17 18:07:17.565656 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:17.565634    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/83cd949f-1579-45be-bbd8-728d45112ef3-config-file\") pod \"limitador-limitador-8588749977-sn8fz\" (UID: \"83cd949f-1579-45be-bbd8-728d45112ef3\") " pod="kuadrant-system/limitador-limitador-8588749977-sn8fz"
Sep 17 18:07:17.573744 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:17.573725    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdnjj\" (UniqueName: \"kubernetes.io/projected/83cd949f-1579-45be-bbd8-728d45112ef3-kube-api-access-jdnjj\") pod \"limitador-limitador-8588749977-sn8fz\" (UID: \"83cd949f-1579-45be-bbd8-728d45112ef3\") " pod="kuadrant-system/limitador-limitador-8588749977-sn8fz"
Sep 17 18:07:17.747506 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:17.747479    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-sn8fz"
Sep 17 18:07:18.023544 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:18.023516    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-sn8fz"]
Sep 17 18:07:18.025478 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:07:18.025451    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83cd949f_1579_45be_bbd8_728d45112ef3.slice/crio-bf81fb00a957966271d2abed5a40d7bcbbab89521e461dcc586d97e04ba28760 WatchSource:0}: Error finding container bf81fb00a957966271d2abed5a40d7bcbbab89521e461dcc586d97e04ba28760: Status 404 returned error can't find the container with id bf81fb00a957966271d2abed5a40d7bcbbab89521e461dcc586d97e04ba28760
Sep 17 18:07:18.465381 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:18.465349    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-4sw49"]
Sep 17 18:07:18.469630 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:18.469609    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-4sw49"
Sep 17 18:07:18.472169 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:18.472141    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qvmw\" (UniqueName: \"kubernetes.io/projected/e36dba87-5206-42d9-9c74-d2f9054d653d-kube-api-access-9qvmw\") pod \"authorino-59c974bd48-4sw49\" (UID: \"e36dba87-5206-42d9-9c74-d2f9054d653d\") " pod="kuadrant-system/authorino-59c974bd48-4sw49"
Sep 17 18:07:18.472285 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:18.472174    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-hrc9j\""
Sep 17 18:07:18.475285 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:18.475261    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-4sw49"]
Sep 17 18:07:18.572910 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:18.572875    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qvmw\" (UniqueName: \"kubernetes.io/projected/e36dba87-5206-42d9-9c74-d2f9054d653d-kube-api-access-9qvmw\") pod \"authorino-59c974bd48-4sw49\" (UID: \"e36dba87-5206-42d9-9c74-d2f9054d653d\") " pod="kuadrant-system/authorino-59c974bd48-4sw49"
Sep 17 18:07:18.582199 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:18.582168    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qvmw\" (UniqueName: \"kubernetes.io/projected/e36dba87-5206-42d9-9c74-d2f9054d653d-kube-api-access-9qvmw\") pod \"authorino-59c974bd48-4sw49\" (UID: \"e36dba87-5206-42d9-9c74-d2f9054d653d\") " pod="kuadrant-system/authorino-59c974bd48-4sw49"
Sep 17 18:07:18.779671 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:18.779581    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-4sw49"
Sep 17 18:07:19.015215 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:19.015181    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-sn8fz" event={"ID":"83cd949f-1579-45be-bbd8-728d45112ef3","Type":"ContainerStarted","Data":"bf81fb00a957966271d2abed5a40d7bcbbab89521e461dcc586d97e04ba28760"}
Sep 17 18:07:19.051508 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:19.051483    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-4sw49"]
Sep 17 18:07:19.053645 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:07:19.053614    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode36dba87_5206_42d9_9c74_d2f9054d653d.slice/crio-5f90687ae52d06fbb2a69360d405a02a10393e0ef265039820882354b0a8b0fe WatchSource:0}: Error finding container 5f90687ae52d06fbb2a69360d405a02a10393e0ef265039820882354b0a8b0fe: Status 404 returned error can't find the container with id 5f90687ae52d06fbb2a69360d405a02a10393e0ef265039820882354b0a8b0fe
Sep 17 18:07:20.022758 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:20.022720    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-4sw49" event={"ID":"e36dba87-5206-42d9-9c74-d2f9054d653d","Type":"ContainerStarted","Data":"5f90687ae52d06fbb2a69360d405a02a10393e0ef265039820882354b0a8b0fe"}
Sep 17 18:07:22.033488 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:22.033454    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-sn8fz" event={"ID":"83cd949f-1579-45be-bbd8-728d45112ef3","Type":"ContainerStarted","Data":"7f63ee1133a8b7000843bef1768f36633305b4b6a8e5b6ebadae6735ee1f3770"}
Sep 17 18:07:22.033926 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:22.033632    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-sn8fz"
Sep 17 18:07:22.034947 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:22.034926    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-4sw49" event={"ID":"e36dba87-5206-42d9-9c74-d2f9054d653d","Type":"ContainerStarted","Data":"333ef501562c19bfd0840438c80eedba49d4b0f6e36098525ca3b1da764b7f46"}
Sep 17 18:07:22.051875 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:22.051816    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-sn8fz" podStartSLOduration=1.116931525 podStartE2EDuration="5.051802121s" totalImagesPullingTime="3.934870596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:07:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:07:22.051258838 +0000 UTC m=+630.141880488" watchObservedRunningTime="2026-09-17 18:07:22.051802121 +0000 UTC m=+630.142423770"
Sep 17 18:07:22.068449 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:22.068404    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-4sw49" podStartSLOduration=1.5798015840000001 podStartE2EDuration="4.068389541s" totalImagesPullingTime="2.488587957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:07:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:07:22.066604928 +0000 UTC m=+630.157226580" watchObservedRunningTime="2026-09-17 18:07:22.068389541 +0000 UTC m=+630.159011191"
Sep 17 18:07:31.788723 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:31.788683    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-sn8fz"]
Sep 17 18:07:31.789193 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:31.788911    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-sn8fz" podUID="83cd949f-1579-45be-bbd8-728d45112ef3" containerName="limitador" containerID="cri-o://7f63ee1133a8b7000843bef1768f36633305b4b6a8e5b6ebadae6735ee1f3770" gracePeriod=30
Sep 17 18:07:31.789726 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:31.789649    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-sn8fz"
Sep 17 18:07:32.341257 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:32.341231    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-sn8fz"
Sep 17 18:07:32.389511 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:32.389434    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdnjj\" (UniqueName: \"kubernetes.io/projected/83cd949f-1579-45be-bbd8-728d45112ef3-kube-api-access-jdnjj\") pod \"83cd949f-1579-45be-bbd8-728d45112ef3\" (UID: \"83cd949f-1579-45be-bbd8-728d45112ef3\") "
Sep 17 18:07:32.389647 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:32.389562    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/83cd949f-1579-45be-bbd8-728d45112ef3-config-file\") pod \"83cd949f-1579-45be-bbd8-728d45112ef3\" (UID: \"83cd949f-1579-45be-bbd8-728d45112ef3\") "
Sep 17 18:07:32.389935 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:32.389896    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83cd949f-1579-45be-bbd8-728d45112ef3-config-file" (OuterVolumeSpecName: "config-file") pod "83cd949f-1579-45be-bbd8-728d45112ef3" (UID: "83cd949f-1579-45be-bbd8-728d45112ef3"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:07:32.391618 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:32.391588    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83cd949f-1579-45be-bbd8-728d45112ef3-kube-api-access-jdnjj" (OuterVolumeSpecName: "kube-api-access-jdnjj") pod "83cd949f-1579-45be-bbd8-728d45112ef3" (UID: "83cd949f-1579-45be-bbd8-728d45112ef3"). InnerVolumeSpecName "kube-api-access-jdnjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:07:32.490295 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:32.490263    2581 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/83cd949f-1579-45be-bbd8-728d45112ef3-config-file\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:07:32.490295 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:32.490292    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdnjj\" (UniqueName: \"kubernetes.io/projected/83cd949f-1579-45be-bbd8-728d45112ef3-kube-api-access-jdnjj\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:07:33.083983 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:33.083947    2581 generic.go:356] "Generic (PLEG): container finished" podID="83cd949f-1579-45be-bbd8-728d45112ef3" containerID="7f63ee1133a8b7000843bef1768f36633305b4b6a8e5b6ebadae6735ee1f3770" exitCode=0
Sep 17 18:07:33.084491 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:33.084011    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-sn8fz"
Sep 17 18:07:33.084491 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:33.084029    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-sn8fz" event={"ID":"83cd949f-1579-45be-bbd8-728d45112ef3","Type":"ContainerDied","Data":"7f63ee1133a8b7000843bef1768f36633305b4b6a8e5b6ebadae6735ee1f3770"}
Sep 17 18:07:33.084491 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:33.084079    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-sn8fz" event={"ID":"83cd949f-1579-45be-bbd8-728d45112ef3","Type":"ContainerDied","Data":"bf81fb00a957966271d2abed5a40d7bcbbab89521e461dcc586d97e04ba28760"}
Sep 17 18:07:33.084491 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:33.084101    2581 scope.go:117] "RemoveContainer" containerID="7f63ee1133a8b7000843bef1768f36633305b4b6a8e5b6ebadae6735ee1f3770"
Sep 17 18:07:33.093241 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:33.093223    2581 scope.go:117] "RemoveContainer" containerID="7f63ee1133a8b7000843bef1768f36633305b4b6a8e5b6ebadae6735ee1f3770"
Sep 17 18:07:33.093484 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:07:33.093464    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f63ee1133a8b7000843bef1768f36633305b4b6a8e5b6ebadae6735ee1f3770\": container with ID starting with 7f63ee1133a8b7000843bef1768f36633305b4b6a8e5b6ebadae6735ee1f3770 not found: ID does not exist" containerID="7f63ee1133a8b7000843bef1768f36633305b4b6a8e5b6ebadae6735ee1f3770"
Sep 17 18:07:33.093521 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:33.093493    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f63ee1133a8b7000843bef1768f36633305b4b6a8e5b6ebadae6735ee1f3770"} err="failed to get container status \"7f63ee1133a8b7000843bef1768f36633305b4b6a8e5b6ebadae6735ee1f3770\": rpc error: code = NotFound desc = could not find container \"7f63ee1133a8b7000843bef1768f36633305b4b6a8e5b6ebadae6735ee1f3770\": container with ID starting with 7f63ee1133a8b7000843bef1768f36633305b4b6a8e5b6ebadae6735ee1f3770 not found: ID does not exist"
Sep 17 18:07:33.107168 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:33.107146    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-sn8fz"]
Sep 17 18:07:33.110710 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:33.110689    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-sn8fz"]
Sep 17 18:07:34.357291 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:34.357261    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83cd949f-1579-45be-bbd8-728d45112ef3" path="/var/lib/kubelet/pods/83cd949f-1579-45be-bbd8-728d45112ef3/volumes"
Sep 17 18:07:41.375847 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:41.375813    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d9898bd99-dv575"]
Sep 17 18:07:41.376262 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:41.376178    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83cd949f-1579-45be-bbd8-728d45112ef3" containerName="limitador"
Sep 17 18:07:41.376262 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:41.376191    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="83cd949f-1579-45be-bbd8-728d45112ef3" containerName="limitador"
Sep 17 18:07:41.376262 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:41.376246    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="83cd949f-1579-45be-bbd8-728d45112ef3" containerName="limitador"
Sep 17 18:07:41.380237 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:41.380217    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-dv575"
Sep 17 18:07:41.383408 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:41.383387    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 17 18:07:41.387754 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:41.387726    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-dv575"]
Sep 17 18:07:41.475106 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:41.475063    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d33c29fb-9b85-4c2c-a933-a75967180a8d-tls-cert\") pod \"authorino-6d9898bd99-dv575\" (UID: \"d33c29fb-9b85-4c2c-a933-a75967180a8d\") " pod="kuadrant-system/authorino-6d9898bd99-dv575"
Sep 17 18:07:41.475275 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:41.475111    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkrfx\" (UniqueName: \"kubernetes.io/projected/d33c29fb-9b85-4c2c-a933-a75967180a8d-kube-api-access-vkrfx\") pod \"authorino-6d9898bd99-dv575\" (UID: \"d33c29fb-9b85-4c2c-a933-a75967180a8d\") " pod="kuadrant-system/authorino-6d9898bd99-dv575"
Sep 17 18:07:41.576472 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:41.576431    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d33c29fb-9b85-4c2c-a933-a75967180a8d-tls-cert\") pod \"authorino-6d9898bd99-dv575\" (UID: \"d33c29fb-9b85-4c2c-a933-a75967180a8d\") " pod="kuadrant-system/authorino-6d9898bd99-dv575"
Sep 17 18:07:41.576472 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:41.576474    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkrfx\" (UniqueName: \"kubernetes.io/projected/d33c29fb-9b85-4c2c-a933-a75967180a8d-kube-api-access-vkrfx\") pod \"authorino-6d9898bd99-dv575\" (UID: \"d33c29fb-9b85-4c2c-a933-a75967180a8d\") " pod="kuadrant-system/authorino-6d9898bd99-dv575"
Sep 17 18:07:41.578892 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:41.578858    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d33c29fb-9b85-4c2c-a933-a75967180a8d-tls-cert\") pod \"authorino-6d9898bd99-dv575\" (UID: \"d33c29fb-9b85-4c2c-a933-a75967180a8d\") " pod="kuadrant-system/authorino-6d9898bd99-dv575"
Sep 17 18:07:41.590420 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:41.590389    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkrfx\" (UniqueName: \"kubernetes.io/projected/d33c29fb-9b85-4c2c-a933-a75967180a8d-kube-api-access-vkrfx\") pod \"authorino-6d9898bd99-dv575\" (UID: \"d33c29fb-9b85-4c2c-a933-a75967180a8d\") " pod="kuadrant-system/authorino-6d9898bd99-dv575"
Sep 17 18:07:41.690318 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:41.690222    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-dv575"
Sep 17 18:07:41.963323 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:41.963290    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-dv575"]
Sep 17 18:07:41.966334 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:07:41.966306    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd33c29fb_9b85_4c2c_a933_a75967180a8d.slice/crio-c261dff65694726a5bb4a4e26a82e609df6e4952e12f15f7cec136fec204c882 WatchSource:0}: Error finding container c261dff65694726a5bb4a4e26a82e609df6e4952e12f15f7cec136fec204c882: Status 404 returned error can't find the container with id c261dff65694726a5bb4a4e26a82e609df6e4952e12f15f7cec136fec204c882
Sep 17 18:07:41.967795 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:41.967774    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:07:42.124400 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:42.124366    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-dv575" event={"ID":"d33c29fb-9b85-4c2c-a933-a75967180a8d","Type":"ContainerStarted","Data":"c261dff65694726a5bb4a4e26a82e609df6e4952e12f15f7cec136fec204c882"}
Sep 17 18:07:43.129832 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:43.129794    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-dv575" event={"ID":"d33c29fb-9b85-4c2c-a933-a75967180a8d","Type":"ContainerStarted","Data":"fe4740fbcef16c20e21bf8aaa010cec528c6064033cd70b51518f6a90dbf7265"}
Sep 17 18:07:43.148737 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:43.148687    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d9898bd99-dv575" podStartSLOduration=1.691321225 podStartE2EDuration="2.148671677s" totalImagesPullingTime="457.350452ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:07:43.147493933 +0000 UTC m=+651.238115582" watchObservedRunningTime="2026-09-17 18:07:43.148671677 +0000 UTC m=+651.239293326"
Sep 17 18:07:43.191865 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:43.191830    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-4sw49"]
Sep 17 18:07:43.192085 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:43.192056    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-4sw49" podUID="e36dba87-5206-42d9-9c74-d2f9054d653d" containerName="authorino" containerID="cri-o://333ef501562c19bfd0840438c80eedba49d4b0f6e36098525ca3b1da764b7f46" gracePeriod=30
Sep 17 18:07:43.451000 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:43.450966    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-4sw49"
Sep 17 18:07:43.594384 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:43.594343    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qvmw\" (UniqueName: \"kubernetes.io/projected/e36dba87-5206-42d9-9c74-d2f9054d653d-kube-api-access-9qvmw\") pod \"e36dba87-5206-42d9-9c74-d2f9054d653d\" (UID: \"e36dba87-5206-42d9-9c74-d2f9054d653d\") "
Sep 17 18:07:43.596406 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:43.596381    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e36dba87-5206-42d9-9c74-d2f9054d653d-kube-api-access-9qvmw" (OuterVolumeSpecName: "kube-api-access-9qvmw") pod "e36dba87-5206-42d9-9c74-d2f9054d653d" (UID: "e36dba87-5206-42d9-9c74-d2f9054d653d"). InnerVolumeSpecName "kube-api-access-9qvmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:07:43.695571 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:43.695485    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qvmw\" (UniqueName: \"kubernetes.io/projected/e36dba87-5206-42d9-9c74-d2f9054d653d-kube-api-access-9qvmw\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:07:44.135811 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:44.135779    2581 generic.go:356] "Generic (PLEG): container finished" podID="e36dba87-5206-42d9-9c74-d2f9054d653d" containerID="333ef501562c19bfd0840438c80eedba49d4b0f6e36098525ca3b1da764b7f46" exitCode=0
Sep 17 18:07:44.137440 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:44.135816    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-4sw49"
Sep 17 18:07:44.137440 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:44.135852    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-4sw49" event={"ID":"e36dba87-5206-42d9-9c74-d2f9054d653d","Type":"ContainerDied","Data":"333ef501562c19bfd0840438c80eedba49d4b0f6e36098525ca3b1da764b7f46"}
Sep 17 18:07:44.137440 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:44.135896    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-4sw49" event={"ID":"e36dba87-5206-42d9-9c74-d2f9054d653d","Type":"ContainerDied","Data":"5f90687ae52d06fbb2a69360d405a02a10393e0ef265039820882354b0a8b0fe"}
Sep 17 18:07:44.137440 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:44.135930    2581 scope.go:117] "RemoveContainer" containerID="333ef501562c19bfd0840438c80eedba49d4b0f6e36098525ca3b1da764b7f46"
Sep 17 18:07:44.147792 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:44.147770    2581 scope.go:117] "RemoveContainer" containerID="333ef501562c19bfd0840438c80eedba49d4b0f6e36098525ca3b1da764b7f46"
Sep 17 18:07:44.148167 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:07:44.148140    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"333ef501562c19bfd0840438c80eedba49d4b0f6e36098525ca3b1da764b7f46\": container with ID starting with 333ef501562c19bfd0840438c80eedba49d4b0f6e36098525ca3b1da764b7f46 not found: ID does not exist" containerID="333ef501562c19bfd0840438c80eedba49d4b0f6e36098525ca3b1da764b7f46"
Sep 17 18:07:44.148256 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:44.148177    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"333ef501562c19bfd0840438c80eedba49d4b0f6e36098525ca3b1da764b7f46"} err="failed to get container status \"333ef501562c19bfd0840438c80eedba49d4b0f6e36098525ca3b1da764b7f46\": rpc error: code = NotFound desc = could not find container \"333ef501562c19bfd0840438c80eedba49d4b0f6e36098525ca3b1da764b7f46\": container with ID starting with 333ef501562c19bfd0840438c80eedba49d4b0f6e36098525ca3b1da764b7f46 not found: ID does not exist"
Sep 17 18:07:44.166708 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:44.166676    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-4sw49"]
Sep 17 18:07:44.172600 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:44.172575    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-4sw49"]
Sep 17 18:07:44.358071 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:07:44.358032    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e36dba87-5206-42d9-9c74-d2f9054d653d" path="/var/lib/kubelet/pods/e36dba87-5206-42d9-9c74-d2f9054d653d/volumes"
Sep 17 18:08:15.072045 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:08:15.072007    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-fjz8s"]
Sep 17 18:08:15.072605 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:08:15.072417    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e36dba87-5206-42d9-9c74-d2f9054d653d" containerName="authorino"
Sep 17 18:08:15.072605 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:08:15.072431    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e36dba87-5206-42d9-9c74-d2f9054d653d" containerName="authorino"
Sep 17 18:08:15.072605 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:08:15.072496    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e36dba87-5206-42d9-9c74-d2f9054d653d" containerName="authorino"
Sep 17 18:08:15.078639 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:08:15.078621    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-fjz8s"
Sep 17 18:08:15.081511 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:08:15.081487    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 18:08:15.081629 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:08:15.081487    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 18:08:15.082373 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:08:15.082348    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-286dp\""
Sep 17 18:08:15.082373 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:08:15.082354    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 18:08:15.091150 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:08:15.091122    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-fjz8s"]
Sep 17 18:08:15.181822 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:08:15.181788    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5d79076f-e5e6-40ea-9ced-376499d7ac0b-data\") pod \"seaweedfs-64568bcd49-fjz8s\" (UID: \"5d79076f-e5e6-40ea-9ced-376499d7ac0b\") " pod="kserve/seaweedfs-64568bcd49-fjz8s"
Sep 17 18:08:15.182023 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:08:15.181881    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm8sx\" (UniqueName: \"kubernetes.io/projected/5d79076f-e5e6-40ea-9ced-376499d7ac0b-kube-api-access-wm8sx\") pod \"seaweedfs-64568bcd49-fjz8s\" (UID: \"5d79076f-e5e6-40ea-9ced-376499d7ac0b\") " pod="kserve/seaweedfs-64568bcd49-fjz8s"
Sep 17 18:08:15.283289 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:08:15.283250    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5d79076f-e5e6-40ea-9ced-376499d7ac0b-data\") pod \"seaweedfs-64568bcd49-fjz8s\" (UID: \"5d79076f-e5e6-40ea-9ced-376499d7ac0b\") " pod="kserve/seaweedfs-64568bcd49-fjz8s"
Sep 17 18:08:15.283476 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:08:15.283327    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm8sx\" (UniqueName: \"kubernetes.io/projected/5d79076f-e5e6-40ea-9ced-376499d7ac0b-kube-api-access-wm8sx\") pod \"seaweedfs-64568bcd49-fjz8s\" (UID: \"5d79076f-e5e6-40ea-9ced-376499d7ac0b\") " pod="kserve/seaweedfs-64568bcd49-fjz8s"
Sep 17 18:08:15.283717 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:08:15.283697    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5d79076f-e5e6-40ea-9ced-376499d7ac0b-data\") pod \"seaweedfs-64568bcd49-fjz8s\" (UID: \"5d79076f-e5e6-40ea-9ced-376499d7ac0b\") " pod="kserve/seaweedfs-64568bcd49-fjz8s"
Sep 17 18:08:15.294473 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:08:15.294444    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm8sx\" (UniqueName: \"kubernetes.io/projected/5d79076f-e5e6-40ea-9ced-376499d7ac0b-kube-api-access-wm8sx\") pod \"seaweedfs-64568bcd49-fjz8s\" (UID: \"5d79076f-e5e6-40ea-9ced-376499d7ac0b\") " pod="kserve/seaweedfs-64568bcd49-fjz8s"
Sep 17 18:08:15.388881 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:08:15.388787    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-fjz8s"
Sep 17 18:08:15.663622 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:08:15.663598    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-fjz8s"]
Sep 17 18:08:15.666161 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:08:15.666131    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d79076f_e5e6_40ea_9ced_376499d7ac0b.slice/crio-18fb7408c32b7857a0bd79f79b048938551b2c2580523b7f40ac55cfdb286e4e WatchSource:0}: Error finding container 18fb7408c32b7857a0bd79f79b048938551b2c2580523b7f40ac55cfdb286e4e: Status 404 returned error can't find the container with id 18fb7408c32b7857a0bd79f79b048938551b2c2580523b7f40ac55cfdb286e4e
Sep 17 18:08:16.272075 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:08:16.272039    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-fjz8s" event={"ID":"5d79076f-e5e6-40ea-9ced-376499d7ac0b","Type":"ContainerStarted","Data":"18fb7408c32b7857a0bd79f79b048938551b2c2580523b7f40ac55cfdb286e4e"}
Sep 17 18:08:19.289023 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:08:19.288647    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-fjz8s" event={"ID":"5d79076f-e5e6-40ea-9ced-376499d7ac0b","Type":"ContainerStarted","Data":"28a025c58de3fc84fdb8a981f213e956b43170f826f4627f6da113e6aa41ba63"}
Sep 17 18:08:19.289439 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:08:19.289062    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-fjz8s"
Sep 17 18:08:19.306725 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:08:19.306676    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-fjz8s" podStartSLOduration=1.238943231 podStartE2EDuration="4.306659058s" totalImagesPullingTime="3.067715827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:19.305496846 +0000 UTC m=+687.396118497" watchObservedRunningTime="2026-09-17 18:08:19.306659058 +0000 UTC m=+687.397280705"
Sep 17 18:08:25.295410 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:08:25.295377    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-fjz8s"
Sep 17 18:11:31.489407 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:31.489371    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs"]
Sep 17 18:11:31.491799 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:31.491776    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs"
Sep 17 18:11:31.496009 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:31.495980    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 17 18:11:31.496158 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:31.496090    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 17 18:11:31.496305 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:31.496283    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 17 18:11:31.496379 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:31.496310    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-thp6p\""
Sep 17 18:11:31.496436 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:31.496424    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-2cjlh\""
Sep 17 18:11:31.503865 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:31.503842    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs"]
Sep 17 18:11:31.513784 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:31.513751    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs\" (UID: \"b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs"
Sep 17 18:11:31.513954 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:31.513805    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dtm5\" (UniqueName: \"kubernetes.io/projected/b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5-kube-api-access-5dtm5\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs\" (UID: \"b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs"
Sep 17 18:11:31.615259 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:31.615213    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs\" (UID: \"b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs"
Sep 17 18:11:31.615459 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:31.615297    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dtm5\" (UniqueName: \"kubernetes.io/projected/b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5-kube-api-access-5dtm5\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs\" (UID: \"b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs"
Sep 17 18:11:31.617781 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:31.617751    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs\" (UID: \"b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs"
Sep 17 18:11:31.623767 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:31.623744    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dtm5\" (UniqueName: \"kubernetes.io/projected/b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5-kube-api-access-5dtm5\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs\" (UID: \"b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs"
Sep 17 18:11:31.803168 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:31.803081    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs"
Sep 17 18:11:32.085130 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:32.085090    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs"]
Sep 17 18:11:32.088329 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:11:32.088286    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ee798f_8e3c_4016_bee6_2ba63b5f6cb5.slice/crio-afd198820f9e095414bc70fa061ea2e8badb99113b634716dca03e3eb1000b37 WatchSource:0}: Error finding container afd198820f9e095414bc70fa061ea2e8badb99113b634716dca03e3eb1000b37: Status 404 returned error can't find the container with id afd198820f9e095414bc70fa061ea2e8badb99113b634716dca03e3eb1000b37
Sep 17 18:11:32.127972 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:32.127930    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs" event={"ID":"b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5","Type":"ContainerStarted","Data":"afd198820f9e095414bc70fa061ea2e8badb99113b634716dca03e3eb1000b37"}
Sep 17 18:11:35.147731 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:35.147655    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs" event={"ID":"b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5","Type":"ContainerStarted","Data":"3bac683e8c5636b8061b0d6c07ac83426806c15d5b690d1ffac86536bbf49036"}
Sep 17 18:11:35.148239 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:35.147809    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs"
Sep 17 18:11:35.165516 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:35.165463    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs" podStartSLOduration=1.20513358 podStartE2EDuration="4.165443836s" totalImagesPullingTime="2.960310256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:11:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:11:35.164413128 +0000 UTC m=+883.255034773" watchObservedRunningTime="2026-09-17 18:11:35.165443836 +0000 UTC m=+883.256065487"
Sep 17 18:11:42.181659 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:42.181625    2581 generic.go:356] "Generic (PLEG): container finished" podID="8421b2a7-bdb3-4dbd-af93-943395f1007f" containerID="d34e47c35aea396e918f89ba89ee72b7412414f8105bfcd2854e29726a9a9fa0" exitCode=0
Sep 17 18:11:42.182148 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:42.181699    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw" event={"ID":"8421b2a7-bdb3-4dbd-af93-943395f1007f","Type":"ContainerDied","Data":"d34e47c35aea396e918f89ba89ee72b7412414f8105bfcd2854e29726a9a9fa0"}
Sep 17 18:11:42.182205 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:42.182158    2581 scope.go:117] "RemoveContainer" containerID="d34e47c35aea396e918f89ba89ee72b7412414f8105bfcd2854e29726a9a9fa0"
Sep 17 18:11:42.183320 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:42.183290    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:11:42.183428 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:42.183357    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:11:42.183428 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:42.183389    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:11:43.188559 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:43.188524    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw" event={"ID":"8421b2a7-bdb3-4dbd-af93-943395f1007f","Type":"ContainerStarted","Data":"f6cd26487f3dc3bf8156656b695d0f407e826990b940524067489590eede843e"}
Sep 17 18:11:44.024158 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:44.024107    2581 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Sep 17 18:11:44.024476 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:44.024420    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw" podUID="8421b2a7-bdb3-4dbd-af93-943395f1007f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Sep 17 18:11:44.024992 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:44.024956    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:11:45.022967 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:45.022856    2581 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Sep 17 18:11:45.023518 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:45.023483    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw" podUID="8421b2a7-bdb3-4dbd-af93-943395f1007f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Sep 17 18:11:46.023779 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:46.023742    2581 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Sep 17 18:11:46.024366 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:46.024334    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw" podUID="8421b2a7-bdb3-4dbd-af93-943395f1007f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Sep 17 18:11:47.023820 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:47.023777    2581 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Sep 17 18:11:47.024349 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:47.023855    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw" podUID="8421b2a7-bdb3-4dbd-af93-943395f1007f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Sep 17 18:11:48.026775 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:48.026743    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:11:48.027224 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:48.027105    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:11:48.027814 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:48.027796    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw"
Sep 17 18:11:52.348120 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:52.348090    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/2.log"
Sep 17 18:11:52.349945 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:11:52.349885    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/2.log"
Sep 17 18:12:06.154573 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:06.154530    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs" podUID="b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:12:06.963984 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:06.963946    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg"]
Sep 17 18:12:06.966916 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:06.966885    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg"
Sep 17 18:12:06.973888 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:06.973862    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 17 18:12:06.974621 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:06.974593    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 17 18:12:06.974714 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:06.974629    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 17 18:12:06.974714 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:06.974665    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-xb4s6\""
Sep 17 18:12:06.975314 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:06.975291    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg"]
Sep 17 18:12:07.037422 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:07.037388    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf861e8a-c916-448d-a0c1-d1f6dace79d1-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-tb8tg\" (UID: \"bf861e8a-c916-448d-a0c1-d1f6dace79d1\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg"
Sep 17 18:12:07.037617 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:07.037453    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf861e8a-c916-448d-a0c1-d1f6dace79d1-home\") pod \"fc-smoke-test-kserve-7f4687897-tb8tg\" (UID: \"bf861e8a-c916-448d-a0c1-d1f6dace79d1\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg"
Sep 17 18:12:07.037617 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:07.037518    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf861e8a-c916-448d-a0c1-d1f6dace79d1-dshm\") pod \"fc-smoke-test-kserve-7f4687897-tb8tg\" (UID: \"bf861e8a-c916-448d-a0c1-d1f6dace79d1\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg"
Sep 17 18:12:07.037617 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:07.037544    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf861e8a-c916-448d-a0c1-d1f6dace79d1-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-tb8tg\" (UID: \"bf861e8a-c916-448d-a0c1-d1f6dace79d1\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg"
Sep 17 18:12:07.037617 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:07.037575    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmvvp\" (UniqueName: \"kubernetes.io/projected/bf861e8a-c916-448d-a0c1-d1f6dace79d1-kube-api-access-pmvvp\") pod \"fc-smoke-test-kserve-7f4687897-tb8tg\" (UID: \"bf861e8a-c916-448d-a0c1-d1f6dace79d1\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg"
Sep 17 18:12:07.037777 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:07.037629    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf861e8a-c916-448d-a0c1-d1f6dace79d1-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-tb8tg\" (UID: \"bf861e8a-c916-448d-a0c1-d1f6dace79d1\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg"
Sep 17 18:12:07.138996 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:07.138954    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf861e8a-c916-448d-a0c1-d1f6dace79d1-home\") pod \"fc-smoke-test-kserve-7f4687897-tb8tg\" (UID: \"bf861e8a-c916-448d-a0c1-d1f6dace79d1\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg"
Sep 17 18:12:07.138996 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:07.138997    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf861e8a-c916-448d-a0c1-d1f6dace79d1-dshm\") pod \"fc-smoke-test-kserve-7f4687897-tb8tg\" (UID: \"bf861e8a-c916-448d-a0c1-d1f6dace79d1\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg"
Sep 17 18:12:07.139243 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:07.139021    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf861e8a-c916-448d-a0c1-d1f6dace79d1-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-tb8tg\" (UID: \"bf861e8a-c916-448d-a0c1-d1f6dace79d1\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg"
Sep 17 18:12:07.139243 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:07.139055    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmvvp\" (UniqueName: \"kubernetes.io/projected/bf861e8a-c916-448d-a0c1-d1f6dace79d1-kube-api-access-pmvvp\") pod \"fc-smoke-test-kserve-7f4687897-tb8tg\" (UID: \"bf861e8a-c916-448d-a0c1-d1f6dace79d1\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg"
Sep 17 18:12:07.139243 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:07.139098    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf861e8a-c916-448d-a0c1-d1f6dace79d1-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-tb8tg\" (UID: \"bf861e8a-c916-448d-a0c1-d1f6dace79d1\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg"
Sep 17 18:12:07.139243 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:07.139163    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf861e8a-c916-448d-a0c1-d1f6dace79d1-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-tb8tg\" (UID: \"bf861e8a-c916-448d-a0c1-d1f6dace79d1\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg"
Sep 17 18:12:07.139441 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:07.139366    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf861e8a-c916-448d-a0c1-d1f6dace79d1-home\") pod \"fc-smoke-test-kserve-7f4687897-tb8tg\" (UID: \"bf861e8a-c916-448d-a0c1-d1f6dace79d1\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg"
Sep 17 18:12:07.139523 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:07.139503    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf861e8a-c916-448d-a0c1-d1f6dace79d1-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-tb8tg\" (UID: \"bf861e8a-c916-448d-a0c1-d1f6dace79d1\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg"
Sep 17 18:12:07.139654 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:07.139517    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf861e8a-c916-448d-a0c1-d1f6dace79d1-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-tb8tg\" (UID: \"bf861e8a-c916-448d-a0c1-d1f6dace79d1\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg"
Sep 17 18:12:07.141252 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:07.141232    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf861e8a-c916-448d-a0c1-d1f6dace79d1-dshm\") pod \"fc-smoke-test-kserve-7f4687897-tb8tg\" (UID: \"bf861e8a-c916-448d-a0c1-d1f6dace79d1\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg"
Sep 17 18:12:07.141627 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:07.141609    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf861e8a-c916-448d-a0c1-d1f6dace79d1-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-tb8tg\" (UID: \"bf861e8a-c916-448d-a0c1-d1f6dace79d1\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg"
Sep 17 18:12:07.152795 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:07.152766    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmvvp\" (UniqueName: \"kubernetes.io/projected/bf861e8a-c916-448d-a0c1-d1f6dace79d1-kube-api-access-pmvvp\") pod \"fc-smoke-test-kserve-7f4687897-tb8tg\" (UID: \"bf861e8a-c916-448d-a0c1-d1f6dace79d1\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg"
Sep 17 18:12:07.177740 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:07.177708    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl"]
Sep 17 18:12:07.180494 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:07.180475    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl"
Sep 17 18:12:07.183547 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:07.183524    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-bfn77\""
Sep 17 18:12:07.190356 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:07.190332    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl"]
Sep 17 18:12:07.278203 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:07.278124    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg"
Sep 17 18:12:07.341384 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:07.341277    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cbb39af-683a-45f6-b84c-944cd66e431e-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl\" (UID: \"1cbb39af-683a-45f6-b84c-944cd66e431e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl"
Sep 17 18:12:07.341384 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:07.341369    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bdgx\" (UniqueName: \"kubernetes.io/projected/1cbb39af-683a-45f6-b84c-944cd66e431e-kube-api-access-2bdgx\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl\" (UID: \"1cbb39af-683a-45f6-b84c-944cd66e431e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl"
Sep 17 18:12:07.442480 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:07.442441    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bdgx\" (UniqueName: \"kubernetes.io/projected/1cbb39af-683a-45f6-b84c-944cd66e431e-kube-api-access-2bdgx\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl\" (UID: \"1cbb39af-683a-45f6-b84c-944cd66e431e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl"
Sep 17 18:12:07.442647 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:07.442507    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cbb39af-683a-45f6-b84c-944cd66e431e-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl\" (UID: \"1cbb39af-683a-45f6-b84c-944cd66e431e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl"
Sep 17 18:12:07.444932 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:07.444895    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cbb39af-683a-45f6-b84c-944cd66e431e-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl\" (UID: \"1cbb39af-683a-45f6-b84c-944cd66e431e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl"
Sep 17 18:12:07.450817 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:07.450796    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bdgx\" (UniqueName: \"kubernetes.io/projected/1cbb39af-683a-45f6-b84c-944cd66e431e-kube-api-access-2bdgx\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl\" (UID: \"1cbb39af-683a-45f6-b84c-944cd66e431e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl"
Sep 17 18:12:07.491862 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:07.491822    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl"
Sep 17 18:12:07.554853 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:07.554825    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg"]
Sep 17 18:12:07.559171 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:12:07.559141    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf861e8a_c916_448d_a0c1_d1f6dace79d1.slice/crio-388d7bdaafe59f0269cc4879232363c069f9e6f6eb9ca75cad883f92031dd0b9 WatchSource:0}: Error finding container 388d7bdaafe59f0269cc4879232363c069f9e6f6eb9ca75cad883f92031dd0b9: Status 404 returned error can't find the container with id 388d7bdaafe59f0269cc4879232363c069f9e6f6eb9ca75cad883f92031dd0b9
Sep 17 18:12:07.771461 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:07.771434    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl"]
Sep 17 18:12:07.773756 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:12:07.773725    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cbb39af_683a_45f6_b84c_944cd66e431e.slice/crio-b848b3055834dcaf778ef04ae91c176af9d3a2dc64b453197494119a0cb39975 WatchSource:0}: Error finding container b848b3055834dcaf778ef04ae91c176af9d3a2dc64b453197494119a0cb39975: Status 404 returned error can't find the container with id b848b3055834dcaf778ef04ae91c176af9d3a2dc64b453197494119a0cb39975
Sep 17 18:12:08.318443 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:08.318394    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl" event={"ID":"1cbb39af-683a-45f6-b84c-944cd66e431e","Type":"ContainerStarted","Data":"b78a252a3aa6c96b5fbc74615463e1e42e1c8eb762fe118b1252e5dba4845b81"}
Sep 17 18:12:08.318995 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:08.318456    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl" event={"ID":"1cbb39af-683a-45f6-b84c-944cd66e431e","Type":"ContainerStarted","Data":"b848b3055834dcaf778ef04ae91c176af9d3a2dc64b453197494119a0cb39975"}
Sep 17 18:12:08.318995 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:08.318519    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl"
Sep 17 18:12:08.320094 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:08.320063    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg" event={"ID":"bf861e8a-c916-448d-a0c1-d1f6dace79d1","Type":"ContainerStarted","Data":"388d7bdaafe59f0269cc4879232363c069f9e6f6eb9ca75cad883f92031dd0b9"}
Sep 17 18:12:08.338450 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:08.338399    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl" podStartSLOduration=1.338377764 podStartE2EDuration="1.338377764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:12:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:08.336869904 +0000 UTC m=+916.427491570" watchObservedRunningTime="2026-09-17 18:12:08.338377764 +0000 UTC m=+916.428999416"
Sep 17 18:12:09.327869 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:09.327832    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg" event={"ID":"bf861e8a-c916-448d-a0c1-d1f6dace79d1","Type":"ContainerStarted","Data":"1b2bf1a14c43340f8df362b9455825f8ca741375a6007ec216bb6e7fe079f895"}
Sep 17 18:12:09.346244 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:09.346168    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg" podStartSLOduration=1.97235034 podStartE2EDuration="3.346149408s" totalImagesPullingTime="1.373799068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:12:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:09.344035283 +0000 UTC m=+917.434656934" watchObservedRunningTime="2026-09-17 18:12:09.346149408 +0000 UTC m=+917.436771055"
Sep 17 18:12:10.279015 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:10.278978    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg"
Sep 17 18:12:16.154008 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:16.153969    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs" podUID="b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:12:17.278723 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:17.278686    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg"
Sep 17 18:12:17.291147 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:17.291122    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg"
Sep 17 18:12:17.376545 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:17.376513    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg"
Sep 17 18:12:19.390194 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:19.390161    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs"]
Sep 17 18:12:19.391863 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:19.391827    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs" podUID="b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:12:29.392263 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:29.392225    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs" podUID="b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:12:34.411085 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:34.411022    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs" podUID="b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5" containerName="main" containerID="cri-o://3bac683e8c5636b8061b0d6c07ac83426806c15d5b690d1ffac86536bbf49036" gracePeriod=45
Sep 17 18:12:39.330208 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:39.330174    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl"
Sep 17 18:12:39.391998 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:39.391963    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs" podUID="b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:12:45.701457 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:45.701403    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl"]
Sep 17 18:12:45.702282 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:45.702255    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg"]
Sep 17 18:12:45.711869 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:12:45.711831    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:12:45.711869 ip-10-0-130-28 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:12:45.711869 ip-10-0-130-28 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:12:45.711869 ip-10-0-130-28 kubenswrapper[2581]:  > containerID="1b2bf1a14c43340f8df362b9455825f8ca741375a6007ec216bb6e7fe079f895" cmd=["/bin/sleep","15"]
Sep 17 18:12:45.712162 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:45.711887    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg" message="" err=<
Sep 17 18:12:45.712162 ip-10-0-130-28 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:12:45.712162 ip-10-0-130-28 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:12:45.712162 ip-10-0-130-28 kubenswrapper[2581]:  >
Sep 17 18:12:45.712162 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:12:45.711928    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:12:45.712162 ip-10-0-130-28 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:12:45.712162 ip-10-0-130-28 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:12:45.712162 ip-10-0-130-28 kubenswrapper[2581]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg" podUID="bf861e8a-c916-448d-a0c1-d1f6dace79d1" containerName="main" containerID="cri-o://1b2bf1a14c43340f8df362b9455825f8ca741375a6007ec216bb6e7fe079f895"
Sep 17 18:12:45.712162 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:45.711976    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg" podUID="bf861e8a-c916-448d-a0c1-d1f6dace79d1" containerName="main" containerID="cri-o://1b2bf1a14c43340f8df362b9455825f8ca741375a6007ec216bb6e7fe079f895" gracePeriod=60
Sep 17 18:12:46.076765 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:46.076738    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg"
Sep 17 18:12:46.193854 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:46.193818    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf861e8a-c916-448d-a0c1-d1f6dace79d1-tls-certs\") pod \"bf861e8a-c916-448d-a0c1-d1f6dace79d1\" (UID: \"bf861e8a-c916-448d-a0c1-d1f6dace79d1\") "
Sep 17 18:12:46.194078 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:46.193926    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf861e8a-c916-448d-a0c1-d1f6dace79d1-tmp-dir\") pod \"bf861e8a-c916-448d-a0c1-d1f6dace79d1\" (UID: \"bf861e8a-c916-448d-a0c1-d1f6dace79d1\") "
Sep 17 18:12:46.194078 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:46.193962    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf861e8a-c916-448d-a0c1-d1f6dace79d1-dshm\") pod \"bf861e8a-c916-448d-a0c1-d1f6dace79d1\" (UID: \"bf861e8a-c916-448d-a0c1-d1f6dace79d1\") "
Sep 17 18:12:46.194078 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:46.193993    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf861e8a-c916-448d-a0c1-d1f6dace79d1-model-cache\") pod \"bf861e8a-c916-448d-a0c1-d1f6dace79d1\" (UID: \"bf861e8a-c916-448d-a0c1-d1f6dace79d1\") "
Sep 17 18:12:46.194078 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:46.194031    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf861e8a-c916-448d-a0c1-d1f6dace79d1-home\") pod \"bf861e8a-c916-448d-a0c1-d1f6dace79d1\" (UID: \"bf861e8a-c916-448d-a0c1-d1f6dace79d1\") "
Sep 17 18:12:46.194078 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:46.194055    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmvvp\" (UniqueName: \"kubernetes.io/projected/bf861e8a-c916-448d-a0c1-d1f6dace79d1-kube-api-access-pmvvp\") pod \"bf861e8a-c916-448d-a0c1-d1f6dace79d1\" (UID: \"bf861e8a-c916-448d-a0c1-d1f6dace79d1\") "
Sep 17 18:12:46.194327 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:46.194203    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf861e8a-c916-448d-a0c1-d1f6dace79d1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bf861e8a-c916-448d-a0c1-d1f6dace79d1" (UID: "bf861e8a-c916-448d-a0c1-d1f6dace79d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:12:46.194327 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:46.194216    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf861e8a-c916-448d-a0c1-d1f6dace79d1-model-cache" (OuterVolumeSpecName: "model-cache") pod "bf861e8a-c916-448d-a0c1-d1f6dace79d1" (UID: "bf861e8a-c916-448d-a0c1-d1f6dace79d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:12:46.194327 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:46.194278    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf861e8a-c916-448d-a0c1-d1f6dace79d1-home" (OuterVolumeSpecName: "home") pod "bf861e8a-c916-448d-a0c1-d1f6dace79d1" (UID: "bf861e8a-c916-448d-a0c1-d1f6dace79d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:12:46.194502 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:46.194430    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf861e8a-c916-448d-a0c1-d1f6dace79d1-model-cache\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:12:46.194502 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:46.194449    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf861e8a-c916-448d-a0c1-d1f6dace79d1-home\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:12:46.194502 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:46.194461    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf861e8a-c916-448d-a0c1-d1f6dace79d1-tmp-dir\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:12:46.196335 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:46.196311    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf861e8a-c916-448d-a0c1-d1f6dace79d1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bf861e8a-c916-448d-a0c1-d1f6dace79d1" (UID: "bf861e8a-c916-448d-a0c1-d1f6dace79d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:12:46.196335 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:46.196327    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf861e8a-c916-448d-a0c1-d1f6dace79d1-dshm" (OuterVolumeSpecName: "dshm") pod "bf861e8a-c916-448d-a0c1-d1f6dace79d1" (UID: "bf861e8a-c916-448d-a0c1-d1f6dace79d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:12:46.196335 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:46.196313    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf861e8a-c916-448d-a0c1-d1f6dace79d1-kube-api-access-pmvvp" (OuterVolumeSpecName: "kube-api-access-pmvvp") pod "bf861e8a-c916-448d-a0c1-d1f6dace79d1" (UID: "bf861e8a-c916-448d-a0c1-d1f6dace79d1"). InnerVolumeSpecName "kube-api-access-pmvvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:12:46.295362 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:46.295283    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf861e8a-c916-448d-a0c1-d1f6dace79d1-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:12:46.295362 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:46.295313    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf861e8a-c916-448d-a0c1-d1f6dace79d1-dshm\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:12:46.295362 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:46.295323    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmvvp\" (UniqueName: \"kubernetes.io/projected/bf861e8a-c916-448d-a0c1-d1f6dace79d1-kube-api-access-pmvvp\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:12:46.504481 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:46.504443    2581 generic.go:356] "Generic (PLEG): container finished" podID="bf861e8a-c916-448d-a0c1-d1f6dace79d1" containerID="1b2bf1a14c43340f8df362b9455825f8ca741375a6007ec216bb6e7fe079f895" exitCode=0
Sep 17 18:12:46.504649 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:46.504494    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg"
Sep 17 18:12:46.504649 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:46.504496    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg" event={"ID":"bf861e8a-c916-448d-a0c1-d1f6dace79d1","Type":"ContainerDied","Data":"1b2bf1a14c43340f8df362b9455825f8ca741375a6007ec216bb6e7fe079f895"}
Sep 17 18:12:46.504649 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:46.504550    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg" event={"ID":"bf861e8a-c916-448d-a0c1-d1f6dace79d1","Type":"ContainerDied","Data":"388d7bdaafe59f0269cc4879232363c069f9e6f6eb9ca75cad883f92031dd0b9"}
Sep 17 18:12:46.504649 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:46.504580    2581 scope.go:117] "RemoveContainer" containerID="1b2bf1a14c43340f8df362b9455825f8ca741375a6007ec216bb6e7fe079f895"
Sep 17 18:12:46.513260 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:46.513243    2581 scope.go:117] "RemoveContainer" containerID="1b2bf1a14c43340f8df362b9455825f8ca741375a6007ec216bb6e7fe079f895"
Sep 17 18:12:46.513552 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:12:46.513533    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b2bf1a14c43340f8df362b9455825f8ca741375a6007ec216bb6e7fe079f895\": container with ID starting with 1b2bf1a14c43340f8df362b9455825f8ca741375a6007ec216bb6e7fe079f895 not found: ID does not exist" containerID="1b2bf1a14c43340f8df362b9455825f8ca741375a6007ec216bb6e7fe079f895"
Sep 17 18:12:46.513606 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:46.513560    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b2bf1a14c43340f8df362b9455825f8ca741375a6007ec216bb6e7fe079f895"} err="failed to get container status \"1b2bf1a14c43340f8df362b9455825f8ca741375a6007ec216bb6e7fe079f895\": rpc error: code = NotFound desc = could not find container \"1b2bf1a14c43340f8df362b9455825f8ca741375a6007ec216bb6e7fe079f895\": container with ID starting with 1b2bf1a14c43340f8df362b9455825f8ca741375a6007ec216bb6e7fe079f895 not found: ID does not exist"
Sep 17 18:12:46.520719 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:46.520695    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg"]
Sep 17 18:12:46.524385 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:46.524364    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-tb8tg"]
Sep 17 18:12:48.358716 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:48.358677    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf861e8a-c916-448d-a0c1-d1f6dace79d1" path="/var/lib/kubelet/pods/bf861e8a-c916-448d-a0c1-d1f6dace79d1/volumes"
Sep 17 18:12:49.329290 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:49.329252    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl" podUID="1cbb39af-683a-45f6-b84c-944cd66e431e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:12:49.392586 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:49.392549    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs" podUID="b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:12:59.329582 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:59.329540    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl" podUID="1cbb39af-683a-45f6-b84c-944cd66e431e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:12:59.391992 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:12:59.391956    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs" podUID="b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:13:00.722338 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:00.722277    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl" podUID="1cbb39af-683a-45f6-b84c-944cd66e431e" containerName="main" containerID="cri-o://b78a252a3aa6c96b5fbc74615463e1e42e1c8eb762fe118b1252e5dba4845b81" gracePeriod=45
Sep 17 18:13:04.590317 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:04.590197    2581 generic.go:356] "Generic (PLEG): container finished" podID="b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5" containerID="3bac683e8c5636b8061b0d6c07ac83426806c15d5b690d1ffac86536bbf49036" exitCode=0
Sep 17 18:13:04.590317 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:04.590281    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs" event={"ID":"b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5","Type":"ContainerDied","Data":"3bac683e8c5636b8061b0d6c07ac83426806c15d5b690d1ffac86536bbf49036"}
Sep 17 18:13:04.655058 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:04.655034    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs"
Sep 17 18:13:04.763038 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:04.763005    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dtm5\" (UniqueName: \"kubernetes.io/projected/b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5-kube-api-access-5dtm5\") pod \"b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5\" (UID: \"b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5\") "
Sep 17 18:13:04.763201 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:04.763062    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5-tls-certs\") pod \"b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5\" (UID: \"b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5\") "
Sep 17 18:13:04.765246 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:04.765210    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5" (UID: "b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:13:04.765246 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:04.765242    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5-kube-api-access-5dtm5" (OuterVolumeSpecName: "kube-api-access-5dtm5") pod "b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5" (UID: "b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5"). InnerVolumeSpecName "kube-api-access-5dtm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:13:04.864525 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:04.864476    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:13:04.864525 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:04.864520    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dtm5\" (UniqueName: \"kubernetes.io/projected/b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5-kube-api-access-5dtm5\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:13:05.596594 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:05.596567    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs"
Sep 17 18:13:05.597011 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:05.596589    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs" event={"ID":"b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5","Type":"ContainerDied","Data":"afd198820f9e095414bc70fa061ea2e8badb99113b634716dca03e3eb1000b37"}
Sep 17 18:13:05.597011 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:05.596649    2581 scope.go:117] "RemoveContainer" containerID="3bac683e8c5636b8061b0d6c07ac83426806c15d5b690d1ffac86536bbf49036"
Sep 17 18:13:05.619096 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:05.619072    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs"]
Sep 17 18:13:05.622006 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:05.621983    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5wh9cs"]
Sep 17 18:13:06.358360 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:06.358321    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5" path="/var/lib/kubelet/pods/b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5/volumes"
Sep 17 18:13:09.329786 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:09.329750    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl" podUID="1cbb39af-683a-45f6-b84c-944cd66e431e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:13:09.330178 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:09.329892    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl"
Sep 17 18:13:19.330214 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:19.330132    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl" podUID="1cbb39af-683a-45f6-b84c-944cd66e431e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:13:22.946519 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:22.946482    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz"]
Sep 17 18:13:22.947020 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:22.946850    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf861e8a-c916-448d-a0c1-d1f6dace79d1" containerName="main"
Sep 17 18:13:22.947020 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:22.946862    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf861e8a-c916-448d-a0c1-d1f6dace79d1" containerName="main"
Sep 17 18:13:22.947020 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:22.946871    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5" containerName="main"
Sep 17 18:13:22.947020 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:22.946876    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5" containerName="main"
Sep 17 18:13:22.947020 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:22.946987    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3ee798f-8e3c-4016-bee6-2ba63b5f6cb5" containerName="main"
Sep 17 18:13:22.947020 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:22.946995    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf861e8a-c916-448d-a0c1-d1f6dace79d1" containerName="main"
Sep 17 18:13:22.948818 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:22.948801    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz"
Sep 17 18:13:22.952793 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:22.952769    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 17 18:13:22.952948 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:22.952793    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 17 18:13:22.952948 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:22.952769    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-wt6nm\""
Sep 17 18:13:22.953123 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:22.953108    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 17 18:13:22.953198 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:22.953182    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-mrfdh\""
Sep 17 18:13:22.958825 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:22.958801    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz"]
Sep 17 18:13:23.026614 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:23.026584    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/5797f9ce-8a5d-4c40-b99e-468797c2af04-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz\" (UID: \"5797f9ce-8a5d-4c40-b99e-468797c2af04\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz"
Sep 17 18:13:23.026614 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:23.026626    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5797f9ce-8a5d-4c40-b99e-468797c2af04-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz\" (UID: \"5797f9ce-8a5d-4c40-b99e-468797c2af04\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz"
Sep 17 18:13:23.026817 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:23.026647    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv4ld\" (UniqueName: \"kubernetes.io/projected/5797f9ce-8a5d-4c40-b99e-468797c2af04-kube-api-access-kv4ld\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz\" (UID: \"5797f9ce-8a5d-4c40-b99e-468797c2af04\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz"
Sep 17 18:13:23.026817 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:23.026756    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5797f9ce-8a5d-4c40-b99e-468797c2af04-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz\" (UID: \"5797f9ce-8a5d-4c40-b99e-468797c2af04\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz"
Sep 17 18:13:23.128054 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:23.128015    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5797f9ce-8a5d-4c40-b99e-468797c2af04-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz\" (UID: \"5797f9ce-8a5d-4c40-b99e-468797c2af04\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz"
Sep 17 18:13:23.128054 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:23.128056    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv4ld\" (UniqueName: \"kubernetes.io/projected/5797f9ce-8a5d-4c40-b99e-468797c2af04-kube-api-access-kv4ld\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz\" (UID: \"5797f9ce-8a5d-4c40-b99e-468797c2af04\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz"
Sep 17 18:13:23.128260 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:23.128124    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5797f9ce-8a5d-4c40-b99e-468797c2af04-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz\" (UID: \"5797f9ce-8a5d-4c40-b99e-468797c2af04\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz"
Sep 17 18:13:23.128260 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:23.128160    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/5797f9ce-8a5d-4c40-b99e-468797c2af04-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz\" (UID: \"5797f9ce-8a5d-4c40-b99e-468797c2af04\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz"
Sep 17 18:13:23.128469 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:23.128454    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/5797f9ce-8a5d-4c40-b99e-468797c2af04-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz\" (UID: \"5797f9ce-8a5d-4c40-b99e-468797c2af04\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz"
Sep 17 18:13:23.128506 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:23.128453    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5797f9ce-8a5d-4c40-b99e-468797c2af04-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz\" (UID: \"5797f9ce-8a5d-4c40-b99e-468797c2af04\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz"
Sep 17 18:13:23.130787 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:23.130761    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5797f9ce-8a5d-4c40-b99e-468797c2af04-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz\" (UID: \"5797f9ce-8a5d-4c40-b99e-468797c2af04\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz"
Sep 17 18:13:23.138674 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:23.138652    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv4ld\" (UniqueName: \"kubernetes.io/projected/5797f9ce-8a5d-4c40-b99e-468797c2af04-kube-api-access-kv4ld\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz\" (UID: \"5797f9ce-8a5d-4c40-b99e-468797c2af04\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz"
Sep 17 18:13:23.259302 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:23.259250    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz"
Sep 17 18:13:23.533707 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:23.533673    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz"]
Sep 17 18:13:23.535724 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:13:23.535691    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5797f9ce_8a5d_4c40_b99e_468797c2af04.slice/crio-1193a555ae010c9e69549fe57e8b7b6aa93d05ed368161b373b2ebdd5b9bc8f9 WatchSource:0}: Error finding container 1193a555ae010c9e69549fe57e8b7b6aa93d05ed368161b373b2ebdd5b9bc8f9: Status 404 returned error can't find the container with id 1193a555ae010c9e69549fe57e8b7b6aa93d05ed368161b373b2ebdd5b9bc8f9
Sep 17 18:13:23.682643 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:23.682607    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz" event={"ID":"5797f9ce-8a5d-4c40-b99e-468797c2af04","Type":"ContainerStarted","Data":"ef090dd8358b10184eb64b41a3b5ae7d7424dbb033543a21f1ae23e8fa3fb186"}
Sep 17 18:13:23.682846 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:23.682673    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz"
Sep 17 18:13:23.682846 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:23.682695    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz" event={"ID":"5797f9ce-8a5d-4c40-b99e-468797c2af04","Type":"ContainerStarted","Data":"1193a555ae010c9e69549fe57e8b7b6aa93d05ed368161b373b2ebdd5b9bc8f9"}
Sep 17 18:13:29.329859 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:29.329824    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl" podUID="1cbb39af-683a-45f6-b84c-944cd66e431e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:13:30.964561 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:30.964536    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl"
Sep 17 18:13:30.987001 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:30.986896    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz" podStartSLOduration=8.986876083 podStartE2EDuration="8.986876083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:13:23.706318814 +0000 UTC m=+991.796940463" watchObservedRunningTime="2026-09-17 18:13:30.986876083 +0000 UTC m=+999.077497734"
Sep 17 18:13:31.098247 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:31.098207    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cbb39af-683a-45f6-b84c-944cd66e431e-tls-certs\") pod \"1cbb39af-683a-45f6-b84c-944cd66e431e\" (UID: \"1cbb39af-683a-45f6-b84c-944cd66e431e\") "
Sep 17 18:13:31.098410 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:31.098260    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bdgx\" (UniqueName: \"kubernetes.io/projected/1cbb39af-683a-45f6-b84c-944cd66e431e-kube-api-access-2bdgx\") pod \"1cbb39af-683a-45f6-b84c-944cd66e431e\" (UID: \"1cbb39af-683a-45f6-b84c-944cd66e431e\") "
Sep 17 18:13:31.100265 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:31.100237    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cbb39af-683a-45f6-b84c-944cd66e431e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1cbb39af-683a-45f6-b84c-944cd66e431e" (UID: "1cbb39af-683a-45f6-b84c-944cd66e431e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:13:31.100379 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:31.100329    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cbb39af-683a-45f6-b84c-944cd66e431e-kube-api-access-2bdgx" (OuterVolumeSpecName: "kube-api-access-2bdgx") pod "1cbb39af-683a-45f6-b84c-944cd66e431e" (UID: "1cbb39af-683a-45f6-b84c-944cd66e431e"). InnerVolumeSpecName "kube-api-access-2bdgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:13:31.199263 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:31.199226    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cbb39af-683a-45f6-b84c-944cd66e431e-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:13:31.199263 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:31.199256    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bdgx\" (UniqueName: \"kubernetes.io/projected/1cbb39af-683a-45f6-b84c-944cd66e431e-kube-api-access-2bdgx\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:13:31.721838 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:31.721797    2581 generic.go:356] "Generic (PLEG): container finished" podID="1cbb39af-683a-45f6-b84c-944cd66e431e" containerID="b78a252a3aa6c96b5fbc74615463e1e42e1c8eb762fe118b1252e5dba4845b81" exitCode=0
Sep 17 18:13:31.721838 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:31.721839    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl"
Sep 17 18:13:31.722102 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:31.721840    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl" event={"ID":"1cbb39af-683a-45f6-b84c-944cd66e431e","Type":"ContainerDied","Data":"b78a252a3aa6c96b5fbc74615463e1e42e1c8eb762fe118b1252e5dba4845b81"}
Sep 17 18:13:31.722102 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:31.721972    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl" event={"ID":"1cbb39af-683a-45f6-b84c-944cd66e431e","Type":"ContainerDied","Data":"b848b3055834dcaf778ef04ae91c176af9d3a2dc64b453197494119a0cb39975"}
Sep 17 18:13:31.722102 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:31.721997    2581 scope.go:117] "RemoveContainer" containerID="b78a252a3aa6c96b5fbc74615463e1e42e1c8eb762fe118b1252e5dba4845b81"
Sep 17 18:13:31.730895 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:31.730878    2581 scope.go:117] "RemoveContainer" containerID="b78a252a3aa6c96b5fbc74615463e1e42e1c8eb762fe118b1252e5dba4845b81"
Sep 17 18:13:31.731176 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:13:31.731159    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b78a252a3aa6c96b5fbc74615463e1e42e1c8eb762fe118b1252e5dba4845b81\": container with ID starting with b78a252a3aa6c96b5fbc74615463e1e42e1c8eb762fe118b1252e5dba4845b81 not found: ID does not exist" containerID="b78a252a3aa6c96b5fbc74615463e1e42e1c8eb762fe118b1252e5dba4845b81"
Sep 17 18:13:31.731213 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:31.731183    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b78a252a3aa6c96b5fbc74615463e1e42e1c8eb762fe118b1252e5dba4845b81"} err="failed to get container status \"b78a252a3aa6c96b5fbc74615463e1e42e1c8eb762fe118b1252e5dba4845b81\": rpc error: code = NotFound desc = could not find container \"b78a252a3aa6c96b5fbc74615463e1e42e1c8eb762fe118b1252e5dba4845b81\": container with ID starting with b78a252a3aa6c96b5fbc74615463e1e42e1c8eb762fe118b1252e5dba4845b81 not found: ID does not exist"
Sep 17 18:13:31.743564 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:31.743542    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl"]
Sep 17 18:13:31.748825 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:31.748803    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-8dcwl"]
Sep 17 18:13:32.358216 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:32.358180    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cbb39af-683a-45f6-b84c-944cd66e431e" path="/var/lib/kubelet/pods/1cbb39af-683a-45f6-b84c-944cd66e431e/volumes"
Sep 17 18:13:39.861926 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:39.861880    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9"]
Sep 17 18:13:39.862294 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:39.862256    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cbb39af-683a-45f6-b84c-944cd66e431e" containerName="main"
Sep 17 18:13:39.862294 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:39.862268    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cbb39af-683a-45f6-b84c-944cd66e431e" containerName="main"
Sep 17 18:13:39.862375 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:39.862346    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cbb39af-683a-45f6-b84c-944cd66e431e" containerName="main"
Sep 17 18:13:39.866373 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:39.866350    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9"
Sep 17 18:13:39.870125 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:39.870102    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-fs2js\""
Sep 17 18:13:39.870274 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:39.870142    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 17 18:13:39.870274 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:39.870168    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 17 18:13:39.870274 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:39.870141    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 17 18:13:39.874468 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:39.874411    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9"]
Sep 17 18:13:39.979211 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:39.979178    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfddc627-3b0f-4e70-a770-3c138a60ba06-home\") pod \"fc-concurrency-test-kserve-b5d44467c-wfps9\" (UID: \"dfddc627-3b0f-4e70-a770-3c138a60ba06\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9"
Sep 17 18:13:39.979389 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:39.979222    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfddc627-3b0f-4e70-a770-3c138a60ba06-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-wfps9\" (UID: \"dfddc627-3b0f-4e70-a770-3c138a60ba06\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9"
Sep 17 18:13:39.979389 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:39.979256    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfddc627-3b0f-4e70-a770-3c138a60ba06-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-wfps9\" (UID: \"dfddc627-3b0f-4e70-a770-3c138a60ba06\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9"
Sep 17 18:13:39.979389 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:39.979289    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfddc627-3b0f-4e70-a770-3c138a60ba06-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-wfps9\" (UID: \"dfddc627-3b0f-4e70-a770-3c138a60ba06\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9"
Sep 17 18:13:39.979389 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:39.979307    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfddc627-3b0f-4e70-a770-3c138a60ba06-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-wfps9\" (UID: \"dfddc627-3b0f-4e70-a770-3c138a60ba06\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9"
Sep 17 18:13:39.979389 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:39.979326    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbrjz\" (UniqueName: \"kubernetes.io/projected/dfddc627-3b0f-4e70-a770-3c138a60ba06-kube-api-access-dbrjz\") pod \"fc-concurrency-test-kserve-b5d44467c-wfps9\" (UID: \"dfddc627-3b0f-4e70-a770-3c138a60ba06\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9"
Sep 17 18:13:40.080141 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:40.080102    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfddc627-3b0f-4e70-a770-3c138a60ba06-home\") pod \"fc-concurrency-test-kserve-b5d44467c-wfps9\" (UID: \"dfddc627-3b0f-4e70-a770-3c138a60ba06\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9"
Sep 17 18:13:40.080346 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:40.080157    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfddc627-3b0f-4e70-a770-3c138a60ba06-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-wfps9\" (UID: \"dfddc627-3b0f-4e70-a770-3c138a60ba06\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9"
Sep 17 18:13:40.080346 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:40.080217    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfddc627-3b0f-4e70-a770-3c138a60ba06-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-wfps9\" (UID: \"dfddc627-3b0f-4e70-a770-3c138a60ba06\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9"
Sep 17 18:13:40.080346 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:40.080282    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfddc627-3b0f-4e70-a770-3c138a60ba06-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-wfps9\" (UID: \"dfddc627-3b0f-4e70-a770-3c138a60ba06\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9"
Sep 17 18:13:40.080346 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:40.080315    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfddc627-3b0f-4e70-a770-3c138a60ba06-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-wfps9\" (UID: \"dfddc627-3b0f-4e70-a770-3c138a60ba06\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9"
Sep 17 18:13:40.080551 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:40.080346    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbrjz\" (UniqueName: \"kubernetes.io/projected/dfddc627-3b0f-4e70-a770-3c138a60ba06-kube-api-access-dbrjz\") pod \"fc-concurrency-test-kserve-b5d44467c-wfps9\" (UID: \"dfddc627-3b0f-4e70-a770-3c138a60ba06\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9"
Sep 17 18:13:40.080681 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:40.080655    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfddc627-3b0f-4e70-a770-3c138a60ba06-home\") pod \"fc-concurrency-test-kserve-b5d44467c-wfps9\" (UID: \"dfddc627-3b0f-4e70-a770-3c138a60ba06\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9"
Sep 17 18:13:40.080750 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:40.080685    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfddc627-3b0f-4e70-a770-3c138a60ba06-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-wfps9\" (UID: \"dfddc627-3b0f-4e70-a770-3c138a60ba06\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9"
Sep 17 18:13:40.080750 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:40.080719    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfddc627-3b0f-4e70-a770-3c138a60ba06-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-wfps9\" (UID: \"dfddc627-3b0f-4e70-a770-3c138a60ba06\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9"
Sep 17 18:13:40.082392 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:40.082373    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfddc627-3b0f-4e70-a770-3c138a60ba06-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-wfps9\" (UID: \"dfddc627-3b0f-4e70-a770-3c138a60ba06\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9"
Sep 17 18:13:40.082666 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:40.082636    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfddc627-3b0f-4e70-a770-3c138a60ba06-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-wfps9\" (UID: \"dfddc627-3b0f-4e70-a770-3c138a60ba06\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9"
Sep 17 18:13:40.089118 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:40.089100    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbrjz\" (UniqueName: \"kubernetes.io/projected/dfddc627-3b0f-4e70-a770-3c138a60ba06-kube-api-access-dbrjz\") pod \"fc-concurrency-test-kserve-b5d44467c-wfps9\" (UID: \"dfddc627-3b0f-4e70-a770-3c138a60ba06\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9"
Sep 17 18:13:40.178398 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:40.178307    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9"
Sep 17 18:13:40.450025 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:40.450002    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9"]
Sep 17 18:13:40.452122 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:13:40.452097    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfddc627_3b0f_4e70_a770_3c138a60ba06.slice/crio-3f1cd42a71a35f88dec687ac44114019f5c570b5cbc3ebc045b81538b402375a WatchSource:0}: Error finding container 3f1cd42a71a35f88dec687ac44114019f5c570b5cbc3ebc045b81538b402375a: Status 404 returned error can't find the container with id 3f1cd42a71a35f88dec687ac44114019f5c570b5cbc3ebc045b81538b402375a
Sep 17 18:13:40.766565 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:40.766523    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9" event={"ID":"dfddc627-3b0f-4e70-a770-3c138a60ba06","Type":"ContainerStarted","Data":"bb488319b8c871a08317b173a1c212ba54bb37eb6d222153ab67a2b5052e4ebd"}
Sep 17 18:13:40.766747 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:40.766619    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9" event={"ID":"dfddc627-3b0f-4e70-a770-3c138a60ba06","Type":"ContainerStarted","Data":"3f1cd42a71a35f88dec687ac44114019f5c570b5cbc3ebc045b81538b402375a"}
Sep 17 18:13:40.793584 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:40.793533    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9" podStartSLOduration=1.793516012 podStartE2EDuration="1.793516012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:13:40.792159749 +0000 UTC m=+1008.882781398" watchObservedRunningTime="2026-09-17 18:13:40.793516012 +0000 UTC m=+1008.884137661"
Sep 17 18:13:41.178923 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:41.178819    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9"
Sep 17 18:13:50.179429 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:50.179388    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9"
Sep 17 18:13:50.191874 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:50.191849    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9"
Sep 17 18:13:50.826919 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:50.826878    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9"
Sep 17 18:13:54.689699 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:54.689666    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz"
Sep 17 18:13:59.625395 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:13:59.625355    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz"]
Sep 17 18:14:04.689763 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:04.689718    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz" podUID="5797f9ce-8a5d-4c40-b99e-468797c2af04" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:14:14.646313 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:14.646265    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz" podUID="5797f9ce-8a5d-4c40-b99e-468797c2af04" containerName="main" containerID="cri-o://ef090dd8358b10184eb64b41a3b5ae7d7424dbb033543a21f1ae23e8fa3fb186" gracePeriod=45
Sep 17 18:14:14.689380 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:14.689344    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz" podUID="5797f9ce-8a5d-4c40-b99e-468797c2af04" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:14:19.906073 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:19.906038    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9"]
Sep 17 18:14:19.918402 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:14:19.918365    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:14:19.918402 ip-10-0-130-28 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:14:19.918402 ip-10-0-130-28 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:14:19.918402 ip-10-0-130-28 kubenswrapper[2581]:  > containerID="bb488319b8c871a08317b173a1c212ba54bb37eb6d222153ab67a2b5052e4ebd" cmd=["/bin/sleep","15"]
Sep 17 18:14:19.918671 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:19.918416    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9" message="" err=<
Sep 17 18:14:19.918671 ip-10-0-130-28 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:14:19.918671 ip-10-0-130-28 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:14:19.918671 ip-10-0-130-28 kubenswrapper[2581]:  >
Sep 17 18:14:19.918671 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:14:19.918429    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:14:19.918671 ip-10-0-130-28 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:14:19.918671 ip-10-0-130-28 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:14:19.918671 ip-10-0-130-28 kubenswrapper[2581]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9" podUID="dfddc627-3b0f-4e70-a770-3c138a60ba06" containerName="main" containerID="cri-o://bb488319b8c871a08317b173a1c212ba54bb37eb6d222153ab67a2b5052e4ebd"
Sep 17 18:14:19.918671 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:19.918478    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9" podUID="dfddc627-3b0f-4e70-a770-3c138a60ba06" containerName="main" containerID="cri-o://bb488319b8c871a08317b173a1c212ba54bb37eb6d222153ab67a2b5052e4ebd" gracePeriod=60
Sep 17 18:14:20.261201 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:20.261171    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9"
Sep 17 18:14:20.353543 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:20.353510    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfddc627-3b0f-4e70-a770-3c138a60ba06-tmp-dir\") pod \"dfddc627-3b0f-4e70-a770-3c138a60ba06\" (UID: \"dfddc627-3b0f-4e70-a770-3c138a60ba06\") "
Sep 17 18:14:20.353695 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:20.353556    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfddc627-3b0f-4e70-a770-3c138a60ba06-tls-certs\") pod \"dfddc627-3b0f-4e70-a770-3c138a60ba06\" (UID: \"dfddc627-3b0f-4e70-a770-3c138a60ba06\") "
Sep 17 18:14:20.353695 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:20.353585    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfddc627-3b0f-4e70-a770-3c138a60ba06-home\") pod \"dfddc627-3b0f-4e70-a770-3c138a60ba06\" (UID: \"dfddc627-3b0f-4e70-a770-3c138a60ba06\") "
Sep 17 18:14:20.353695 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:20.353607    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfddc627-3b0f-4e70-a770-3c138a60ba06-model-cache\") pod \"dfddc627-3b0f-4e70-a770-3c138a60ba06\" (UID: \"dfddc627-3b0f-4e70-a770-3c138a60ba06\") "
Sep 17 18:14:20.353695 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:20.353668    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbrjz\" (UniqueName: \"kubernetes.io/projected/dfddc627-3b0f-4e70-a770-3c138a60ba06-kube-api-access-dbrjz\") pod \"dfddc627-3b0f-4e70-a770-3c138a60ba06\" (UID: \"dfddc627-3b0f-4e70-a770-3c138a60ba06\") "
Sep 17 18:14:20.353941 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:20.353704    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfddc627-3b0f-4e70-a770-3c138a60ba06-dshm\") pod \"dfddc627-3b0f-4e70-a770-3c138a60ba06\" (UID: \"dfddc627-3b0f-4e70-a770-3c138a60ba06\") "
Sep 17 18:14:20.353941 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:20.353842    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfddc627-3b0f-4e70-a770-3c138a60ba06-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dfddc627-3b0f-4e70-a770-3c138a60ba06" (UID: "dfddc627-3b0f-4e70-a770-3c138a60ba06"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:20.353941 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:20.353840    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfddc627-3b0f-4e70-a770-3c138a60ba06-home" (OuterVolumeSpecName: "home") pod "dfddc627-3b0f-4e70-a770-3c138a60ba06" (UID: "dfddc627-3b0f-4e70-a770-3c138a60ba06"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:20.354098 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:20.353945    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfddc627-3b0f-4e70-a770-3c138a60ba06-model-cache" (OuterVolumeSpecName: "model-cache") pod "dfddc627-3b0f-4e70-a770-3c138a60ba06" (UID: "dfddc627-3b0f-4e70-a770-3c138a60ba06"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:20.354153 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:20.354100    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfddc627-3b0f-4e70-a770-3c138a60ba06-home\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:14:20.354153 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:20.354119    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfddc627-3b0f-4e70-a770-3c138a60ba06-model-cache\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:14:20.354153 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:20.354135    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfddc627-3b0f-4e70-a770-3c138a60ba06-tmp-dir\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:14:20.356011 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:20.355980    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfddc627-3b0f-4e70-a770-3c138a60ba06-dshm" (OuterVolumeSpecName: "dshm") pod "dfddc627-3b0f-4e70-a770-3c138a60ba06" (UID: "dfddc627-3b0f-4e70-a770-3c138a60ba06"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:20.356137 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:20.356095    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfddc627-3b0f-4e70-a770-3c138a60ba06-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dfddc627-3b0f-4e70-a770-3c138a60ba06" (UID: "dfddc627-3b0f-4e70-a770-3c138a60ba06"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:14:20.356137 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:20.356122    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfddc627-3b0f-4e70-a770-3c138a60ba06-kube-api-access-dbrjz" (OuterVolumeSpecName: "kube-api-access-dbrjz") pod "dfddc627-3b0f-4e70-a770-3c138a60ba06" (UID: "dfddc627-3b0f-4e70-a770-3c138a60ba06"). InnerVolumeSpecName "kube-api-access-dbrjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:14:20.455478 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:20.455451    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbrjz\" (UniqueName: \"kubernetes.io/projected/dfddc627-3b0f-4e70-a770-3c138a60ba06-kube-api-access-dbrjz\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:14:20.455478 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:20.455476    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfddc627-3b0f-4e70-a770-3c138a60ba06-dshm\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:14:20.455686 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:20.455485    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfddc627-3b0f-4e70-a770-3c138a60ba06-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:14:20.954635 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:20.954600    2581 generic.go:356] "Generic (PLEG): container finished" podID="dfddc627-3b0f-4e70-a770-3c138a60ba06" containerID="bb488319b8c871a08317b173a1c212ba54bb37eb6d222153ab67a2b5052e4ebd" exitCode=0
Sep 17 18:14:20.955098 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:20.954644    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9"
Sep 17 18:14:20.955098 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:20.954685    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9" event={"ID":"dfddc627-3b0f-4e70-a770-3c138a60ba06","Type":"ContainerDied","Data":"bb488319b8c871a08317b173a1c212ba54bb37eb6d222153ab67a2b5052e4ebd"}
Sep 17 18:14:20.955098 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:20.954740    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9" event={"ID":"dfddc627-3b0f-4e70-a770-3c138a60ba06","Type":"ContainerDied","Data":"3f1cd42a71a35f88dec687ac44114019f5c570b5cbc3ebc045b81538b402375a"}
Sep 17 18:14:20.955098 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:20.954767    2581 scope.go:117] "RemoveContainer" containerID="bb488319b8c871a08317b173a1c212ba54bb37eb6d222153ab67a2b5052e4ebd"
Sep 17 18:14:20.967951 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:20.967926    2581 scope.go:117] "RemoveContainer" containerID="bb488319b8c871a08317b173a1c212ba54bb37eb6d222153ab67a2b5052e4ebd"
Sep 17 18:14:20.968292 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:14:20.968266    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb488319b8c871a08317b173a1c212ba54bb37eb6d222153ab67a2b5052e4ebd\": container with ID starting with bb488319b8c871a08317b173a1c212ba54bb37eb6d222153ab67a2b5052e4ebd not found: ID does not exist" containerID="bb488319b8c871a08317b173a1c212ba54bb37eb6d222153ab67a2b5052e4ebd"
Sep 17 18:14:20.968341 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:20.968301    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb488319b8c871a08317b173a1c212ba54bb37eb6d222153ab67a2b5052e4ebd"} err="failed to get container status \"bb488319b8c871a08317b173a1c212ba54bb37eb6d222153ab67a2b5052e4ebd\": rpc error: code = NotFound desc = could not find container \"bb488319b8c871a08317b173a1c212ba54bb37eb6d222153ab67a2b5052e4ebd\": container with ID starting with bb488319b8c871a08317b173a1c212ba54bb37eb6d222153ab67a2b5052e4ebd not found: ID does not exist"
Sep 17 18:14:20.977724 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:20.977701    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9"]
Sep 17 18:14:20.981061 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:20.981036    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-wfps9"]
Sep 17 18:14:22.359104 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:22.359069    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfddc627-3b0f-4e70-a770-3c138a60ba06" path="/var/lib/kubelet/pods/dfddc627-3b0f-4e70-a770-3c138a60ba06/volumes"
Sep 17 18:14:24.689641 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:24.689597    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz" podUID="5797f9ce-8a5d-4c40-b99e-468797c2af04" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:14:24.690110 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:24.689720    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz"
Sep 17 18:14:34.689958 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:34.689887    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz" podUID="5797f9ce-8a5d-4c40-b99e-468797c2af04" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:14:44.688332 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:14:44.688307    2581 logging.go:55] [core] [Channel #101 SubChannel #102]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.33:9003", ServerName: "10.133.0.33:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.33:9003: connect: connection refused"
Sep 17 18:14:44.885121 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:44.885095    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz"
Sep 17 18:14:44.979341 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:44.979307    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/5797f9ce-8a5d-4c40-b99e-468797c2af04-cachi2-cache\") pod \"5797f9ce-8a5d-4c40-b99e-468797c2af04\" (UID: \"5797f9ce-8a5d-4c40-b99e-468797c2af04\") "
Sep 17 18:14:44.979533 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:44.979362    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5797f9ce-8a5d-4c40-b99e-468797c2af04-tokenizer-cache\") pod \"5797f9ce-8a5d-4c40-b99e-468797c2af04\" (UID: \"5797f9ce-8a5d-4c40-b99e-468797c2af04\") "
Sep 17 18:14:44.979533 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:44.979404    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kv4ld\" (UniqueName: \"kubernetes.io/projected/5797f9ce-8a5d-4c40-b99e-468797c2af04-kube-api-access-kv4ld\") pod \"5797f9ce-8a5d-4c40-b99e-468797c2af04\" (UID: \"5797f9ce-8a5d-4c40-b99e-468797c2af04\") "
Sep 17 18:14:44.979533 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:44.979433    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5797f9ce-8a5d-4c40-b99e-468797c2af04-tls-certs\") pod \"5797f9ce-8a5d-4c40-b99e-468797c2af04\" (UID: \"5797f9ce-8a5d-4c40-b99e-468797c2af04\") "
Sep 17 18:14:44.979750 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:44.979617    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5797f9ce-8a5d-4c40-b99e-468797c2af04-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "5797f9ce-8a5d-4c40-b99e-468797c2af04" (UID: "5797f9ce-8a5d-4c40-b99e-468797c2af04"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:44.979750 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:44.979637    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5797f9ce-8a5d-4c40-b99e-468797c2af04-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "5797f9ce-8a5d-4c40-b99e-468797c2af04" (UID: "5797f9ce-8a5d-4c40-b99e-468797c2af04"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:44.979750 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:44.979729    2581 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/5797f9ce-8a5d-4c40-b99e-468797c2af04-cachi2-cache\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:14:44.979750 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:44.979741    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/5797f9ce-8a5d-4c40-b99e-468797c2af04-tokenizer-cache\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:14:44.981844 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:44.981814    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5797f9ce-8a5d-4c40-b99e-468797c2af04-kube-api-access-kv4ld" (OuterVolumeSpecName: "kube-api-access-kv4ld") pod "5797f9ce-8a5d-4c40-b99e-468797c2af04" (UID: "5797f9ce-8a5d-4c40-b99e-468797c2af04"). InnerVolumeSpecName "kube-api-access-kv4ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:14:44.981844 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:44.981833    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5797f9ce-8a5d-4c40-b99e-468797c2af04-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5797f9ce-8a5d-4c40-b99e-468797c2af04" (UID: "5797f9ce-8a5d-4c40-b99e-468797c2af04"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:14:45.066602 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:45.066523    2581 generic.go:356] "Generic (PLEG): container finished" podID="5797f9ce-8a5d-4c40-b99e-468797c2af04" containerID="ef090dd8358b10184eb64b41a3b5ae7d7424dbb033543a21f1ae23e8fa3fb186" exitCode=0
Sep 17 18:14:45.066602 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:45.066567    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz"
Sep 17 18:14:45.066771 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:45.066613    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz" event={"ID":"5797f9ce-8a5d-4c40-b99e-468797c2af04","Type":"ContainerDied","Data":"ef090dd8358b10184eb64b41a3b5ae7d7424dbb033543a21f1ae23e8fa3fb186"}
Sep 17 18:14:45.066771 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:45.066682    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz" event={"ID":"5797f9ce-8a5d-4c40-b99e-468797c2af04","Type":"ContainerDied","Data":"1193a555ae010c9e69549fe57e8b7b6aa93d05ed368161b373b2ebdd5b9bc8f9"}
Sep 17 18:14:45.066771 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:45.066713    2581 scope.go:117] "RemoveContainer" containerID="ef090dd8358b10184eb64b41a3b5ae7d7424dbb033543a21f1ae23e8fa3fb186"
Sep 17 18:14:45.075677 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:45.075659    2581 scope.go:117] "RemoveContainer" containerID="ef090dd8358b10184eb64b41a3b5ae7d7424dbb033543a21f1ae23e8fa3fb186"
Sep 17 18:14:45.075965 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:14:45.075945    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef090dd8358b10184eb64b41a3b5ae7d7424dbb033543a21f1ae23e8fa3fb186\": container with ID starting with ef090dd8358b10184eb64b41a3b5ae7d7424dbb033543a21f1ae23e8fa3fb186 not found: ID does not exist" containerID="ef090dd8358b10184eb64b41a3b5ae7d7424dbb033543a21f1ae23e8fa3fb186"
Sep 17 18:14:45.076022 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:45.075972    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef090dd8358b10184eb64b41a3b5ae7d7424dbb033543a21f1ae23e8fa3fb186"} err="failed to get container status \"ef090dd8358b10184eb64b41a3b5ae7d7424dbb033543a21f1ae23e8fa3fb186\": rpc error: code = NotFound desc = could not find container \"ef090dd8358b10184eb64b41a3b5ae7d7424dbb033543a21f1ae23e8fa3fb186\": container with ID starting with ef090dd8358b10184eb64b41a3b5ae7d7424dbb033543a21f1ae23e8fa3fb186 not found: ID does not exist"
Sep 17 18:14:45.080717 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:45.080698    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv4ld\" (UniqueName: \"kubernetes.io/projected/5797f9ce-8a5d-4c40-b99e-468797c2af04-kube-api-access-kv4ld\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:14:45.080768 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:45.080720    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5797f9ce-8a5d-4c40-b99e-468797c2af04-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:14:45.087983 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:45.087958    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz"]
Sep 17 18:14:45.090074 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:45.090054    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz"]
Sep 17 18:14:45.688249 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:45.688206    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcs4tbz" podUID="5797f9ce-8a5d-4c40-b99e-468797c2af04" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.133.0.33:9003\" within 1s: context deadline exceeded"
Sep 17 18:14:46.357591 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:46.357559    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5797f9ce-8a5d-4c40-b99e-468797c2af04" path="/var/lib/kubelet/pods/5797f9ce-8a5d-4c40-b99e-468797c2af04/volumes"
Sep 17 18:14:52.897507 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:52.897463    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"]
Sep 17 18:14:52.898118 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:52.898095    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfddc627-3b0f-4e70-a770-3c138a60ba06" containerName="main"
Sep 17 18:14:52.898118 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:52.898119    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfddc627-3b0f-4e70-a770-3c138a60ba06" containerName="main"
Sep 17 18:14:52.898272 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:52.898154    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5797f9ce-8a5d-4c40-b99e-468797c2af04" containerName="main"
Sep 17 18:14:52.898272 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:52.898163    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5797f9ce-8a5d-4c40-b99e-468797c2af04" containerName="main"
Sep 17 18:14:52.898272 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:52.898257    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5797f9ce-8a5d-4c40-b99e-468797c2af04" containerName="main"
Sep 17 18:14:52.898272 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:52.898269    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfddc627-3b0f-4e70-a770-3c138a60ba06" containerName="main"
Sep 17 18:14:52.903161 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:52.903132    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"
Sep 17 18:14:52.906273 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:52.906251    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 17 18:14:52.907367 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:52.907346    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 17 18:14:52.907525 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:52.907500    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-2nmgt\""
Sep 17 18:14:52.907658 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:52.907605    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-pvxrm\""
Sep 17 18:14:52.907859 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:52.907745    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 17 18:14:52.911693 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:52.911375    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"]
Sep 17 18:14:52.947571 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:52.947530    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn7cs\" (UniqueName: \"kubernetes.io/projected/15718008-2428-4fb1-bd86-f2826369e535-kube-api-access-qn7cs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv\" (UID: \"15718008-2428-4fb1-bd86-f2826369e535\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"
Sep 17 18:14:52.947751 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:52.947585    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15718008-2428-4fb1-bd86-f2826369e535-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv\" (UID: \"15718008-2428-4fb1-bd86-f2826369e535\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"
Sep 17 18:14:52.947751 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:52.947605    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15718008-2428-4fb1-bd86-f2826369e535-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv\" (UID: \"15718008-2428-4fb1-bd86-f2826369e535\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"
Sep 17 18:14:52.947751 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:52.947659    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15718008-2428-4fb1-bd86-f2826369e535-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv\" (UID: \"15718008-2428-4fb1-bd86-f2826369e535\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"
Sep 17 18:14:52.947751 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:52.947722    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15718008-2428-4fb1-bd86-f2826369e535-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv\" (UID: \"15718008-2428-4fb1-bd86-f2826369e535\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"
Sep 17 18:14:52.947933 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:52.947754    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15718008-2428-4fb1-bd86-f2826369e535-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv\" (UID: \"15718008-2428-4fb1-bd86-f2826369e535\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"
Sep 17 18:14:53.048462 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:53.048420    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn7cs\" (UniqueName: \"kubernetes.io/projected/15718008-2428-4fb1-bd86-f2826369e535-kube-api-access-qn7cs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv\" (UID: \"15718008-2428-4fb1-bd86-f2826369e535\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"
Sep 17 18:14:53.048659 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:53.048479    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15718008-2428-4fb1-bd86-f2826369e535-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv\" (UID: \"15718008-2428-4fb1-bd86-f2826369e535\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"
Sep 17 18:14:53.048659 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:53.048508    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15718008-2428-4fb1-bd86-f2826369e535-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv\" (UID: \"15718008-2428-4fb1-bd86-f2826369e535\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"
Sep 17 18:14:53.048659 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:53.048563    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15718008-2428-4fb1-bd86-f2826369e535-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv\" (UID: \"15718008-2428-4fb1-bd86-f2826369e535\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"
Sep 17 18:14:53.048659 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:53.048609    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15718008-2428-4fb1-bd86-f2826369e535-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv\" (UID: \"15718008-2428-4fb1-bd86-f2826369e535\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"
Sep 17 18:14:53.048659 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:53.048649    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15718008-2428-4fb1-bd86-f2826369e535-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv\" (UID: \"15718008-2428-4fb1-bd86-f2826369e535\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"
Sep 17 18:14:53.049000 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:53.048973    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15718008-2428-4fb1-bd86-f2826369e535-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv\" (UID: \"15718008-2428-4fb1-bd86-f2826369e535\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"
Sep 17 18:14:53.049000 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:53.048993    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15718008-2428-4fb1-bd86-f2826369e535-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv\" (UID: \"15718008-2428-4fb1-bd86-f2826369e535\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"
Sep 17 18:14:53.049115 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:53.049054    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15718008-2428-4fb1-bd86-f2826369e535-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv\" (UID: \"15718008-2428-4fb1-bd86-f2826369e535\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"
Sep 17 18:14:53.050869 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:53.050849    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15718008-2428-4fb1-bd86-f2826369e535-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv\" (UID: \"15718008-2428-4fb1-bd86-f2826369e535\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"
Sep 17 18:14:53.051067 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:53.051047    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15718008-2428-4fb1-bd86-f2826369e535-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv\" (UID: \"15718008-2428-4fb1-bd86-f2826369e535\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"
Sep 17 18:14:53.056659 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:53.056634    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn7cs\" (UniqueName: \"kubernetes.io/projected/15718008-2428-4fb1-bd86-f2826369e535-kube-api-access-qn7cs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv\" (UID: \"15718008-2428-4fb1-bd86-f2826369e535\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"
Sep 17 18:14:53.113279 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:53.113246    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx"]
Sep 17 18:14:53.116638 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:53.116615    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx"
Sep 17 18:14:53.118973 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:53.118953    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-dl8mh\""
Sep 17 18:14:53.124877 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:53.124852    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx"]
Sep 17 18:14:53.150021 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:53.149928    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56d5664f-6361-498f-843e-ff2cbed0d58c-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx\" (UID: \"56d5664f-6361-498f-843e-ff2cbed0d58c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx"
Sep 17 18:14:53.150021 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:53.149980    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h6kh\" (UniqueName: \"kubernetes.io/projected/56d5664f-6361-498f-843e-ff2cbed0d58c-kube-api-access-7h6kh\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx\" (UID: \"56d5664f-6361-498f-843e-ff2cbed0d58c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx"
Sep 17 18:14:53.214482 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:53.214439    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"
Sep 17 18:14:53.250784 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:53.250745    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h6kh\" (UniqueName: \"kubernetes.io/projected/56d5664f-6361-498f-843e-ff2cbed0d58c-kube-api-access-7h6kh\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx\" (UID: \"56d5664f-6361-498f-843e-ff2cbed0d58c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx"
Sep 17 18:14:53.250963 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:53.250842    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56d5664f-6361-498f-843e-ff2cbed0d58c-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx\" (UID: \"56d5664f-6361-498f-843e-ff2cbed0d58c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx"
Sep 17 18:14:53.253474 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:53.253417    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56d5664f-6361-498f-843e-ff2cbed0d58c-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx\" (UID: \"56d5664f-6361-498f-843e-ff2cbed0d58c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx"
Sep 17 18:14:53.258319 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:53.258289    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h6kh\" (UniqueName: \"kubernetes.io/projected/56d5664f-6361-498f-843e-ff2cbed0d58c-kube-api-access-7h6kh\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx\" (UID: \"56d5664f-6361-498f-843e-ff2cbed0d58c\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx"
Sep 17 18:14:53.429540 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:53.429450    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx"
Sep 17 18:14:53.490997 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:53.490819    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"]
Sep 17 18:14:53.493350 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:14:53.493323    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15718008_2428_4fb1_bd86_f2826369e535.slice/crio-c5c59a4a449b578d97ffd11b592b588f0c07b01ceeaa168518d8242ef62fcb94 WatchSource:0}: Error finding container c5c59a4a449b578d97ffd11b592b588f0c07b01ceeaa168518d8242ef62fcb94: Status 404 returned error can't find the container with id c5c59a4a449b578d97ffd11b592b588f0c07b01ceeaa168518d8242ef62fcb94
Sep 17 18:14:53.495404 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:53.495390    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:14:53.704113 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:53.704087    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx"]
Sep 17 18:14:53.706818 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:14:53.706786    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d5664f_6361_498f_843e_ff2cbed0d58c.slice/crio-b8ee8e9d36f541a7501f1bde36db1c5123418964953e57c026ecd9a26d5ecf6b WatchSource:0}: Error finding container b8ee8e9d36f541a7501f1bde36db1c5123418964953e57c026ecd9a26d5ecf6b: Status 404 returned error can't find the container with id b8ee8e9d36f541a7501f1bde36db1c5123418964953e57c026ecd9a26d5ecf6b
Sep 17 18:14:54.118539 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:54.118445    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx" event={"ID":"56d5664f-6361-498f-843e-ff2cbed0d58c","Type":"ContainerStarted","Data":"b5c8f476733fe2134f8cde1f8568332629af0a866593c2055dd0aa655c37c437"}
Sep 17 18:14:54.118539 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:54.118513    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx" event={"ID":"56d5664f-6361-498f-843e-ff2cbed0d58c","Type":"ContainerStarted","Data":"b8ee8e9d36f541a7501f1bde36db1c5123418964953e57c026ecd9a26d5ecf6b"}
Sep 17 18:14:54.119079 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:54.118566    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx"
Sep 17 18:14:54.120793 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:54.120756    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv" event={"ID":"15718008-2428-4fb1-bd86-f2826369e535","Type":"ContainerStarted","Data":"c5c59a4a449b578d97ffd11b592b588f0c07b01ceeaa168518d8242ef62fcb94"}
Sep 17 18:14:54.135669 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:54.135613    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx" podStartSLOduration=1.13559458 podStartE2EDuration="1.13559458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:14:54.134589544 +0000 UTC m=+1082.225211195" watchObservedRunningTime="2026-09-17 18:14:54.13559458 +0000 UTC m=+1082.226216231"
Sep 17 18:14:56.134530 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:56.134487    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv" event={"ID":"15718008-2428-4fb1-bd86-f2826369e535","Type":"ContainerStarted","Data":"0aba293d29045bf96630f48b6b7b43dfd9645bc1d1f62145ea65a7fb682e4198"}
Sep 17 18:14:56.134958 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:56.134589    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"
Sep 17 18:14:57.140848 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:57.140811    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv" event={"ID":"15718008-2428-4fb1-bd86-f2826369e535","Type":"ContainerStarted","Data":"c6d1152f4b15d4d4161047a70ec9e0db84960e49ec457bd91277630bbc2e3165"}
Sep 17 18:14:57.162896 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:57.162839    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv" podStartSLOduration=3.414077871 podStartE2EDuration="5.162822949s" totalImagesPullingTime="1.748745078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:14:57.161472614 +0000 UTC m=+1085.252094267" watchObservedRunningTime="2026-09-17 18:14:57.162822949 +0000 UTC m=+1085.253444593"
Sep 17 18:14:57.215400 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:14:57.215363    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"
Sep 17 18:15:03.215238 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:03.215197    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"
Sep 17 18:15:03.227598 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:03.227565    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"
Sep 17 18:15:04.184682 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:04.184652    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"
Sep 17 18:15:05.191483 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:05.191451    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"
Sep 17 18:15:25.129225 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:25.129194    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx"
Sep 17 18:15:29.460268 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.460236    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"]
Sep 17 18:15:29.471864 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.471839    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:15:29.475931 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.475649    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 17 18:15:29.476127 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.476100    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-ksszc\""
Sep 17 18:15:29.476365 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.476347    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Sep 17 18:15:29.476586 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.476568    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 17 18:15:29.478270 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.478248    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"]
Sep 17 18:15:29.493447 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.493418    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntl9z\" (UniqueName: \"kubernetes.io/projected/66425f3f-4776-4a2f-b579-1a2af3486c0e-kube-api-access-ntl9z\") pod \"router-gateway-1-openshift-default-7dcbd489bd-mwldr\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:15:29.493588 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.493465    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66425f3f-4776-4a2f-b579-1a2af3486c0e-credential-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-mwldr\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:15:29.493588 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.493544    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66425f3f-4776-4a2f-b579-1a2af3486c0e-istio-envoy\") pod \"router-gateway-1-openshift-default-7dcbd489bd-mwldr\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:15:29.493922 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.493643    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66425f3f-4776-4a2f-b579-1a2af3486c0e-workload-certs\") pod \"router-gateway-1-openshift-default-7dcbd489bd-mwldr\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:15:29.493922 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.493683    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66425f3f-4776-4a2f-b579-1a2af3486c0e-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7dcbd489bd-mwldr\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:15:29.493922 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.493757    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66425f3f-4776-4a2f-b579-1a2af3486c0e-istio-podinfo\") pod \"router-gateway-1-openshift-default-7dcbd489bd-mwldr\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:15:29.493922 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.493788    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66425f3f-4776-4a2f-b579-1a2af3486c0e-workload-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-mwldr\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:15:29.493922 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.493824    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66425f3f-4776-4a2f-b579-1a2af3486c0e-istio-token\") pod \"router-gateway-1-openshift-default-7dcbd489bd-mwldr\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:15:29.493922 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.493887    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66425f3f-4776-4a2f-b579-1a2af3486c0e-istio-data\") pod \"router-gateway-1-openshift-default-7dcbd489bd-mwldr\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:15:29.595243 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.595205    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66425f3f-4776-4a2f-b579-1a2af3486c0e-istio-data\") pod \"router-gateway-1-openshift-default-7dcbd489bd-mwldr\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:15:29.595400 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.595266    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntl9z\" (UniqueName: \"kubernetes.io/projected/66425f3f-4776-4a2f-b579-1a2af3486c0e-kube-api-access-ntl9z\") pod \"router-gateway-1-openshift-default-7dcbd489bd-mwldr\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:15:29.595400 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.595289    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66425f3f-4776-4a2f-b579-1a2af3486c0e-credential-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-mwldr\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:15:29.595400 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.595307    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66425f3f-4776-4a2f-b579-1a2af3486c0e-istio-envoy\") pod \"router-gateway-1-openshift-default-7dcbd489bd-mwldr\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:15:29.595400 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.595352    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66425f3f-4776-4a2f-b579-1a2af3486c0e-workload-certs\") pod \"router-gateway-1-openshift-default-7dcbd489bd-mwldr\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:15:29.595400 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.595376    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66425f3f-4776-4a2f-b579-1a2af3486c0e-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7dcbd489bd-mwldr\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:15:29.595647 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.595419    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66425f3f-4776-4a2f-b579-1a2af3486c0e-istio-podinfo\") pod \"router-gateway-1-openshift-default-7dcbd489bd-mwldr\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:15:29.595647 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.595445    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66425f3f-4776-4a2f-b579-1a2af3486c0e-workload-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-mwldr\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:15:29.595647 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.595520    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66425f3f-4776-4a2f-b579-1a2af3486c0e-istio-token\") pod \"router-gateway-1-openshift-default-7dcbd489bd-mwldr\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:15:29.595647 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.595616    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66425f3f-4776-4a2f-b579-1a2af3486c0e-istio-data\") pod \"router-gateway-1-openshift-default-7dcbd489bd-mwldr\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:15:29.595860 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.595837    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66425f3f-4776-4a2f-b579-1a2af3486c0e-workload-certs\") pod \"router-gateway-1-openshift-default-7dcbd489bd-mwldr\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:15:29.595932 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.595877    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66425f3f-4776-4a2f-b579-1a2af3486c0e-credential-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-mwldr\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:15:29.595993 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.595937    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66425f3f-4776-4a2f-b579-1a2af3486c0e-workload-socket\") pod \"router-gateway-1-openshift-default-7dcbd489bd-mwldr\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:15:29.596220 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.596204    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66425f3f-4776-4a2f-b579-1a2af3486c0e-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-7dcbd489bd-mwldr\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:15:29.597497 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.597476    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66425f3f-4776-4a2f-b579-1a2af3486c0e-istio-envoy\") pod \"router-gateway-1-openshift-default-7dcbd489bd-mwldr\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:15:29.597958 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.597940    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66425f3f-4776-4a2f-b579-1a2af3486c0e-istio-podinfo\") pod \"router-gateway-1-openshift-default-7dcbd489bd-mwldr\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:15:29.603255 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.603237    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66425f3f-4776-4a2f-b579-1a2af3486c0e-istio-token\") pod \"router-gateway-1-openshift-default-7dcbd489bd-mwldr\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:15:29.603503 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.603487    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntl9z\" (UniqueName: \"kubernetes.io/projected/66425f3f-4776-4a2f-b579-1a2af3486c0e-kube-api-access-ntl9z\") pod \"router-gateway-1-openshift-default-7dcbd489bd-mwldr\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:15:29.786419 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:29.786317    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:15:30.062795 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:30.062770    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"]
Sep 17 18:15:30.065195 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:15:30.065165    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66425f3f_4776_4a2f_b579_1a2af3486c0e.slice/crio-357825389dac9bf2c293985907b6fb1a0f0762f83591d11f9de49deae9dda307 WatchSource:0}: Error finding container 357825389dac9bf2c293985907b6fb1a0f0762f83591d11f9de49deae9dda307: Status 404 returned error can't find the container with id 357825389dac9bf2c293985907b6fb1a0f0762f83591d11f9de49deae9dda307
Sep 17 18:15:30.067185 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:30.067153    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:15:30.067276 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:30.067234    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:15:30.067340 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:30.067281    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:15:30.315815 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:30.315719    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr" event={"ID":"66425f3f-4776-4a2f-b579-1a2af3486c0e","Type":"ContainerStarted","Data":"492ab76904e05b7c9f9ca1866eca53057666bbb28bca0a5068666cc014ad9762"}
Sep 17 18:15:30.315815 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:30.315811    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr" event={"ID":"66425f3f-4776-4a2f-b579-1a2af3486c0e","Type":"ContainerStarted","Data":"357825389dac9bf2c293985907b6fb1a0f0762f83591d11f9de49deae9dda307"}
Sep 17 18:15:30.333504 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:30.333457    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr" podStartSLOduration=1.333441415 podStartE2EDuration="1.333441415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:15:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:15:30.332596438 +0000 UTC m=+1118.423218125" watchObservedRunningTime="2026-09-17 18:15:30.333441415 +0000 UTC m=+1118.424063068"
Sep 17 18:15:30.787202 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:30.787161    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:15:31.792573 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:31.792538    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:15:32.325168 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:32.325138    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:15:32.326125 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:32.326103    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:15:58.181610 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:58.181574    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"]
Sep 17 18:15:58.182184 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:15:58.181783    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr" podUID="66425f3f-4776-4a2f-b579-1a2af3486c0e" containerName="istio-proxy" containerID="cri-o://492ab76904e05b7c9f9ca1866eca53057666bbb28bca0a5068666cc014ad9762" gracePeriod=30
Sep 17 18:16:00.824332 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:00.824308    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:16:00.984664 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:00.984630    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntl9z\" (UniqueName: \"kubernetes.io/projected/66425f3f-4776-4a2f-b579-1a2af3486c0e-kube-api-access-ntl9z\") pod \"66425f3f-4776-4a2f-b579-1a2af3486c0e\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") "
Sep 17 18:16:00.984849 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:00.984684    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66425f3f-4776-4a2f-b579-1a2af3486c0e-istio-podinfo\") pod \"66425f3f-4776-4a2f-b579-1a2af3486c0e\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") "
Sep 17 18:16:00.984849 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:00.984706    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66425f3f-4776-4a2f-b579-1a2af3486c0e-istiod-ca-cert\") pod \"66425f3f-4776-4a2f-b579-1a2af3486c0e\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") "
Sep 17 18:16:00.984849 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:00.984733    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66425f3f-4776-4a2f-b579-1a2af3486c0e-workload-certs\") pod \"66425f3f-4776-4a2f-b579-1a2af3486c0e\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") "
Sep 17 18:16:00.984849 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:00.984774    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66425f3f-4776-4a2f-b579-1a2af3486c0e-workload-socket\") pod \"66425f3f-4776-4a2f-b579-1a2af3486c0e\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") "
Sep 17 18:16:00.984849 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:00.984810    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66425f3f-4776-4a2f-b579-1a2af3486c0e-credential-socket\") pod \"66425f3f-4776-4a2f-b579-1a2af3486c0e\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") "
Sep 17 18:16:00.985140 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:00.984849    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66425f3f-4776-4a2f-b579-1a2af3486c0e-istio-data\") pod \"66425f3f-4776-4a2f-b579-1a2af3486c0e\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") "
Sep 17 18:16:00.985140 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:00.984953    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66425f3f-4776-4a2f-b579-1a2af3486c0e-istio-envoy\") pod \"66425f3f-4776-4a2f-b579-1a2af3486c0e\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") "
Sep 17 18:16:00.985140 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:00.984976    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66425f3f-4776-4a2f-b579-1a2af3486c0e-istio-token\") pod \"66425f3f-4776-4a2f-b579-1a2af3486c0e\" (UID: \"66425f3f-4776-4a2f-b579-1a2af3486c0e\") "
Sep 17 18:16:00.985140 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:00.985112    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66425f3f-4776-4a2f-b579-1a2af3486c0e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "66425f3f-4776-4a2f-b579-1a2af3486c0e" (UID: "66425f3f-4776-4a2f-b579-1a2af3486c0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:16:00.985348 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:00.985158    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66425f3f-4776-4a2f-b579-1a2af3486c0e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "66425f3f-4776-4a2f-b579-1a2af3486c0e" (UID: "66425f3f-4776-4a2f-b579-1a2af3486c0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:16:00.985348 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:00.985192    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66425f3f-4776-4a2f-b579-1a2af3486c0e-istio-data" (OuterVolumeSpecName: "istio-data") pod "66425f3f-4776-4a2f-b579-1a2af3486c0e" (UID: "66425f3f-4776-4a2f-b579-1a2af3486c0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:16:00.985348 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:00.985256    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66425f3f-4776-4a2f-b579-1a2af3486c0e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "66425f3f-4776-4a2f-b579-1a2af3486c0e" (UID: "66425f3f-4776-4a2f-b579-1a2af3486c0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:16:00.985489 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:00.985384    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66425f3f-4776-4a2f-b579-1a2af3486c0e-istio-data\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:16:00.985489 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:00.985406    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66425f3f-4776-4a2f-b579-1a2af3486c0e-istiod-ca-cert\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:16:00.985489 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:00.985421    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66425f3f-4776-4a2f-b579-1a2af3486c0e-workload-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:16:00.985489 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:00.985419    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66425f3f-4776-4a2f-b579-1a2af3486c0e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "66425f3f-4776-4a2f-b579-1a2af3486c0e" (UID: "66425f3f-4776-4a2f-b579-1a2af3486c0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:16:00.985489 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:00.985433    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66425f3f-4776-4a2f-b579-1a2af3486c0e-workload-socket\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:16:00.987185 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:00.987158    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66425f3f-4776-4a2f-b579-1a2af3486c0e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "66425f3f-4776-4a2f-b579-1a2af3486c0e" (UID: "66425f3f-4776-4a2f-b579-1a2af3486c0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 18:16:00.987185 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:00.987174    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66425f3f-4776-4a2f-b579-1a2af3486c0e-istio-token" (OuterVolumeSpecName: "istio-token") pod "66425f3f-4776-4a2f-b579-1a2af3486c0e" (UID: "66425f3f-4776-4a2f-b579-1a2af3486c0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:16:00.987340 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:00.987217    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66425f3f-4776-4a2f-b579-1a2af3486c0e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "66425f3f-4776-4a2f-b579-1a2af3486c0e" (UID: "66425f3f-4776-4a2f-b579-1a2af3486c0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:16:00.987340 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:00.987255    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66425f3f-4776-4a2f-b579-1a2af3486c0e-kube-api-access-ntl9z" (OuterVolumeSpecName: "kube-api-access-ntl9z") pod "66425f3f-4776-4a2f-b579-1a2af3486c0e" (UID: "66425f3f-4776-4a2f-b579-1a2af3486c0e"). InnerVolumeSpecName "kube-api-access-ntl9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:16:01.086353 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:01.086319    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntl9z\" (UniqueName: \"kubernetes.io/projected/66425f3f-4776-4a2f-b579-1a2af3486c0e-kube-api-access-ntl9z\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:16:01.086353 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:01.086352    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66425f3f-4776-4a2f-b579-1a2af3486c0e-istio-podinfo\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:16:01.086556 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:01.086365    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66425f3f-4776-4a2f-b579-1a2af3486c0e-credential-socket\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:16:01.086556 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:01.086379    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66425f3f-4776-4a2f-b579-1a2af3486c0e-istio-envoy\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:16:01.086556 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:01.086390    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66425f3f-4776-4a2f-b579-1a2af3486c0e-istio-token\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:16:01.470937 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:01.470883    2581 generic.go:356] "Generic (PLEG): container finished" podID="66425f3f-4776-4a2f-b579-1a2af3486c0e" containerID="492ab76904e05b7c9f9ca1866eca53057666bbb28bca0a5068666cc014ad9762" exitCode=0
Sep 17 18:16:01.471114 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:01.470968    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"
Sep 17 18:16:01.471114 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:01.470976    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr" event={"ID":"66425f3f-4776-4a2f-b579-1a2af3486c0e","Type":"ContainerDied","Data":"492ab76904e05b7c9f9ca1866eca53057666bbb28bca0a5068666cc014ad9762"}
Sep 17 18:16:01.471114 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:01.471034    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr" event={"ID":"66425f3f-4776-4a2f-b579-1a2af3486c0e","Type":"ContainerDied","Data":"357825389dac9bf2c293985907b6fb1a0f0762f83591d11f9de49deae9dda307"}
Sep 17 18:16:01.471114 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:01.471058    2581 scope.go:117] "RemoveContainer" containerID="492ab76904e05b7c9f9ca1866eca53057666bbb28bca0a5068666cc014ad9762"
Sep 17 18:16:01.480832 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:01.480814    2581 scope.go:117] "RemoveContainer" containerID="492ab76904e05b7c9f9ca1866eca53057666bbb28bca0a5068666cc014ad9762"
Sep 17 18:16:01.481108 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:16:01.481087    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"492ab76904e05b7c9f9ca1866eca53057666bbb28bca0a5068666cc014ad9762\": container with ID starting with 492ab76904e05b7c9f9ca1866eca53057666bbb28bca0a5068666cc014ad9762 not found: ID does not exist" containerID="492ab76904e05b7c9f9ca1866eca53057666bbb28bca0a5068666cc014ad9762"
Sep 17 18:16:01.481170 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:01.481116    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"492ab76904e05b7c9f9ca1866eca53057666bbb28bca0a5068666cc014ad9762"} err="failed to get container status \"492ab76904e05b7c9f9ca1866eca53057666bbb28bca0a5068666cc014ad9762\": rpc error: code = NotFound desc = could not find container \"492ab76904e05b7c9f9ca1866eca53057666bbb28bca0a5068666cc014ad9762\": container with ID starting with 492ab76904e05b7c9f9ca1866eca53057666bbb28bca0a5068666cc014ad9762 not found: ID does not exist"
Sep 17 18:16:01.492137 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:01.492113    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"]
Sep 17 18:16:01.496078 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:01.496049    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-7dcbd489bd-mwldr"]
Sep 17 18:16:02.358218 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:02.358186    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66425f3f-4776-4a2f-b579-1a2af3486c0e" path="/var/lib/kubelet/pods/66425f3f-4776-4a2f-b579-1a2af3486c0e/volumes"
Sep 17 18:16:03.347875 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:03.347830    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm"]
Sep 17 18:16:03.348404 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:03.348367    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66425f3f-4776-4a2f-b579-1a2af3486c0e" containerName="istio-proxy"
Sep 17 18:16:03.348404 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:03.348395    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="66425f3f-4776-4a2f-b579-1a2af3486c0e" containerName="istio-proxy"
Sep 17 18:16:03.348569 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:03.348543    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="66425f3f-4776-4a2f-b579-1a2af3486c0e" containerName="istio-proxy"
Sep 17 18:16:03.351067 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:03.351049    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm"
Sep 17 18:16:03.354536 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:03.354510    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-987bk\""
Sep 17 18:16:03.354701 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:03.354575    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 17 18:16:03.354701 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:03.354575    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 17 18:16:03.354701 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:03.354679    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 17 18:16:03.354916 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:03.354699    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-p6n8x\""
Sep 17 18:16:03.358731 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:03.358709    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm"]
Sep 17 18:16:03.509520 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:03.509489    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz5xq\" (UniqueName: \"kubernetes.io/projected/b6bc9e69-b6b3-4c16-be12-853b6f9b5764-kube-api-access-gz5xq\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm\" (UID: \"b6bc9e69-b6b3-4c16-be12-853b6f9b5764\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm"
Sep 17 18:16:03.509715 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:03.509576    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6bc9e69-b6b3-4c16-be12-853b6f9b5764-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm\" (UID: \"b6bc9e69-b6b3-4c16-be12-853b6f9b5764\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm"
Sep 17 18:16:03.611148 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:03.611044    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz5xq\" (UniqueName: \"kubernetes.io/projected/b6bc9e69-b6b3-4c16-be12-853b6f9b5764-kube-api-access-gz5xq\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm\" (UID: \"b6bc9e69-b6b3-4c16-be12-853b6f9b5764\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm"
Sep 17 18:16:03.611148 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:03.611124    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6bc9e69-b6b3-4c16-be12-853b6f9b5764-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm\" (UID: \"b6bc9e69-b6b3-4c16-be12-853b6f9b5764\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm"
Sep 17 18:16:03.613716 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:03.613682    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6bc9e69-b6b3-4c16-be12-853b6f9b5764-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm\" (UID: \"b6bc9e69-b6b3-4c16-be12-853b6f9b5764\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm"
Sep 17 18:16:03.618622 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:03.618603    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz5xq\" (UniqueName: \"kubernetes.io/projected/b6bc9e69-b6b3-4c16-be12-853b6f9b5764-kube-api-access-gz5xq\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm\" (UID: \"b6bc9e69-b6b3-4c16-be12-853b6f9b5764\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm"
Sep 17 18:16:03.662518 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:03.662481    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm"
Sep 17 18:16:03.932574 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:03.932546    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm"]
Sep 17 18:16:03.934376 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:16:03.934347    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6bc9e69_b6b3_4c16_be12_853b6f9b5764.slice/crio-5614cdca5089073aedbf10530c3bc9a6d5c836fa4bbb614746d8de3e3e302398 WatchSource:0}: Error finding container 5614cdca5089073aedbf10530c3bc9a6d5c836fa4bbb614746d8de3e3e302398: Status 404 returned error can't find the container with id 5614cdca5089073aedbf10530c3bc9a6d5c836fa4bbb614746d8de3e3e302398
Sep 17 18:16:04.489111 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:04.489075    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm" event={"ID":"b6bc9e69-b6b3-4c16-be12-853b6f9b5764","Type":"ContainerStarted","Data":"527783aca1b7667f00ecdec4ccac254717a9c9bd28447fa8805b1df24ba2c319"}
Sep 17 18:16:04.489505 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:04.489128    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm" event={"ID":"b6bc9e69-b6b3-4c16-be12-853b6f9b5764","Type":"ContainerStarted","Data":"5614cdca5089073aedbf10530c3bc9a6d5c836fa4bbb614746d8de3e3e302398"}
Sep 17 18:16:04.489505 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:04.489159    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm"
Sep 17 18:16:04.508849 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:04.508799    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm" podStartSLOduration=1.5087826930000001 podStartE2EDuration="1.508782693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:04.507267266 +0000 UTC m=+1152.597888916" watchObservedRunningTime="2026-09-17 18:16:04.508782693 +0000 UTC m=+1152.599404343"
Sep 17 18:16:35.495623 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:35.495592    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm"
Sep 17 18:16:40.394159 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:40.394120    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"]
Sep 17 18:16:40.398690 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:40.398650    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx"]
Sep 17 18:16:40.404470 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:16:40.404431    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:16:40.404470 ip-10-0-130-28 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:16:40.404470 ip-10-0-130-28 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:16:40.404470 ip-10-0-130-28 kubenswrapper[2581]:  > containerID="c6d1152f4b15d4d4161047a70ec9e0db84960e49ec457bd91277630bbc2e3165" cmd=["/bin/sleep","15"]
Sep 17 18:16:40.404704 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:40.404494    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv" message="" err=<
Sep 17 18:16:40.404704 ip-10-0-130-28 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:16:40.404704 ip-10-0-130-28 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:16:40.404704 ip-10-0-130-28 kubenswrapper[2581]:  >
Sep 17 18:16:40.404704 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:16:40.404513    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:16:40.404704 ip-10-0-130-28 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:16:40.404704 ip-10-0-130-28 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:16:40.404704 ip-10-0-130-28 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv" podUID="15718008-2428-4fb1-bd86-f2826369e535" containerName="main" containerID="cri-o://c6d1152f4b15d4d4161047a70ec9e0db84960e49ec457bd91277630bbc2e3165"
Sep 17 18:16:40.404704 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:40.404558    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv" podUID="15718008-2428-4fb1-bd86-f2826369e535" containerName="main" containerID="cri-o://c6d1152f4b15d4d4161047a70ec9e0db84960e49ec457bd91277630bbc2e3165" gracePeriod=60
Sep 17 18:16:40.620993 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:40.620962    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv" podUID="15718008-2428-4fb1-bd86-f2826369e535" containerName="llm-d-routing-sidecar" containerID="cri-o://0aba293d29045bf96630f48b6b7b43dfd9645bc1d1f62145ea65a7fb682e4198" gracePeriod=60
Sep 17 18:16:40.662385 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:40.662323    2581 generic.go:356] "Generic (PLEG): container finished" podID="15718008-2428-4fb1-bd86-f2826369e535" containerID="c6d1152f4b15d4d4161047a70ec9e0db84960e49ec457bd91277630bbc2e3165" exitCode=0
Sep 17 18:16:40.662475 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:40.662377    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv" event={"ID":"15718008-2428-4fb1-bd86-f2826369e535","Type":"ContainerDied","Data":"c6d1152f4b15d4d4161047a70ec9e0db84960e49ec457bd91277630bbc2e3165"}
Sep 17 18:16:40.876602 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:40.876573    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"
Sep 17 18:16:41.057456 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:41.057421    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15718008-2428-4fb1-bd86-f2826369e535-model-cache\") pod \"15718008-2428-4fb1-bd86-f2826369e535\" (UID: \"15718008-2428-4fb1-bd86-f2826369e535\") "
Sep 17 18:16:41.057663 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:41.057513    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15718008-2428-4fb1-bd86-f2826369e535-dshm\") pod \"15718008-2428-4fb1-bd86-f2826369e535\" (UID: \"15718008-2428-4fb1-bd86-f2826369e535\") "
Sep 17 18:16:41.057663 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:41.057548    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qn7cs\" (UniqueName: \"kubernetes.io/projected/15718008-2428-4fb1-bd86-f2826369e535-kube-api-access-qn7cs\") pod \"15718008-2428-4fb1-bd86-f2826369e535\" (UID: \"15718008-2428-4fb1-bd86-f2826369e535\") "
Sep 17 18:16:41.057663 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:41.057583    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15718008-2428-4fb1-bd86-f2826369e535-tls-certs\") pod \"15718008-2428-4fb1-bd86-f2826369e535\" (UID: \"15718008-2428-4fb1-bd86-f2826369e535\") "
Sep 17 18:16:41.057663 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:41.057607    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15718008-2428-4fb1-bd86-f2826369e535-home\") pod \"15718008-2428-4fb1-bd86-f2826369e535\" (UID: \"15718008-2428-4fb1-bd86-f2826369e535\") "
Sep 17 18:16:41.057663 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:41.057634    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15718008-2428-4fb1-bd86-f2826369e535-tmp-dir\") pod \"15718008-2428-4fb1-bd86-f2826369e535\" (UID: \"15718008-2428-4fb1-bd86-f2826369e535\") "
Sep 17 18:16:41.057997 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:41.057697    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15718008-2428-4fb1-bd86-f2826369e535-model-cache" (OuterVolumeSpecName: "model-cache") pod "15718008-2428-4fb1-bd86-f2826369e535" (UID: "15718008-2428-4fb1-bd86-f2826369e535"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:16:41.057997 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:41.057880    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15718008-2428-4fb1-bd86-f2826369e535-home" (OuterVolumeSpecName: "home") pod "15718008-2428-4fb1-bd86-f2826369e535" (UID: "15718008-2428-4fb1-bd86-f2826369e535"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:16:41.057997 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:41.057951    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15718008-2428-4fb1-bd86-f2826369e535-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "15718008-2428-4fb1-bd86-f2826369e535" (UID: "15718008-2428-4fb1-bd86-f2826369e535"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:16:41.057997 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:41.057986    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15718008-2428-4fb1-bd86-f2826369e535-model-cache\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:16:41.059860 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:41.059836    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15718008-2428-4fb1-bd86-f2826369e535-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "15718008-2428-4fb1-bd86-f2826369e535" (UID: "15718008-2428-4fb1-bd86-f2826369e535"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:16:41.059860 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:41.059835    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15718008-2428-4fb1-bd86-f2826369e535-kube-api-access-qn7cs" (OuterVolumeSpecName: "kube-api-access-qn7cs") pod "15718008-2428-4fb1-bd86-f2826369e535" (UID: "15718008-2428-4fb1-bd86-f2826369e535"). InnerVolumeSpecName "kube-api-access-qn7cs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:16:41.060037 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:41.059839    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15718008-2428-4fb1-bd86-f2826369e535-dshm" (OuterVolumeSpecName: "dshm") pod "15718008-2428-4fb1-bd86-f2826369e535" (UID: "15718008-2428-4fb1-bd86-f2826369e535"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:16:41.158744 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:41.158707    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15718008-2428-4fb1-bd86-f2826369e535-dshm\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:16:41.158934 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:41.158756    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn7cs\" (UniqueName: \"kubernetes.io/projected/15718008-2428-4fb1-bd86-f2826369e535-kube-api-access-qn7cs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:16:41.158934 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:41.158768    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15718008-2428-4fb1-bd86-f2826369e535-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:16:41.158934 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:41.158780    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15718008-2428-4fb1-bd86-f2826369e535-home\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:16:41.158934 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:41.158788    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15718008-2428-4fb1-bd86-f2826369e535-tmp-dir\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:16:41.668879 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:41.668840    2581 generic.go:356] "Generic (PLEG): container finished" podID="15718008-2428-4fb1-bd86-f2826369e535" containerID="0aba293d29045bf96630f48b6b7b43dfd9645bc1d1f62145ea65a7fb682e4198" exitCode=0
Sep 17 18:16:41.669286 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:41.668931    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"
Sep 17 18:16:41.669286 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:41.668931    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv" event={"ID":"15718008-2428-4fb1-bd86-f2826369e535","Type":"ContainerDied","Data":"0aba293d29045bf96630f48b6b7b43dfd9645bc1d1f62145ea65a7fb682e4198"}
Sep 17 18:16:41.669286 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:41.668989    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv" event={"ID":"15718008-2428-4fb1-bd86-f2826369e535","Type":"ContainerDied","Data":"c5c59a4a449b578d97ffd11b592b588f0c07b01ceeaa168518d8242ef62fcb94"}
Sep 17 18:16:41.669286 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:41.669012    2581 scope.go:117] "RemoveContainer" containerID="c6d1152f4b15d4d4161047a70ec9e0db84960e49ec457bd91277630bbc2e3165"
Sep 17 18:16:41.678293 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:41.678271    2581 scope.go:117] "RemoveContainer" containerID="0aba293d29045bf96630f48b6b7b43dfd9645bc1d1f62145ea65a7fb682e4198"
Sep 17 18:16:41.686091 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:41.686070    2581 scope.go:117] "RemoveContainer" containerID="c6d1152f4b15d4d4161047a70ec9e0db84960e49ec457bd91277630bbc2e3165"
Sep 17 18:16:41.686380 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:16:41.686357    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6d1152f4b15d4d4161047a70ec9e0db84960e49ec457bd91277630bbc2e3165\": container with ID starting with c6d1152f4b15d4d4161047a70ec9e0db84960e49ec457bd91277630bbc2e3165 not found: ID does not exist" containerID="c6d1152f4b15d4d4161047a70ec9e0db84960e49ec457bd91277630bbc2e3165"
Sep 17 18:16:41.686433 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:41.686389    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6d1152f4b15d4d4161047a70ec9e0db84960e49ec457bd91277630bbc2e3165"} err="failed to get container status \"c6d1152f4b15d4d4161047a70ec9e0db84960e49ec457bd91277630bbc2e3165\": rpc error: code = NotFound desc = could not find container \"c6d1152f4b15d4d4161047a70ec9e0db84960e49ec457bd91277630bbc2e3165\": container with ID starting with c6d1152f4b15d4d4161047a70ec9e0db84960e49ec457bd91277630bbc2e3165 not found: ID does not exist"
Sep 17 18:16:41.686433 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:41.686407    2581 scope.go:117] "RemoveContainer" containerID="0aba293d29045bf96630f48b6b7b43dfd9645bc1d1f62145ea65a7fb682e4198"
Sep 17 18:16:41.686647 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:16:41.686633    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aba293d29045bf96630f48b6b7b43dfd9645bc1d1f62145ea65a7fb682e4198\": container with ID starting with 0aba293d29045bf96630f48b6b7b43dfd9645bc1d1f62145ea65a7fb682e4198 not found: ID does not exist" containerID="0aba293d29045bf96630f48b6b7b43dfd9645bc1d1f62145ea65a7fb682e4198"
Sep 17 18:16:41.686687 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:41.686651    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aba293d29045bf96630f48b6b7b43dfd9645bc1d1f62145ea65a7fb682e4198"} err="failed to get container status \"0aba293d29045bf96630f48b6b7b43dfd9645bc1d1f62145ea65a7fb682e4198\": rpc error: code = NotFound desc = could not find container \"0aba293d29045bf96630f48b6b7b43dfd9645bc1d1f62145ea65a7fb682e4198\": container with ID starting with 0aba293d29045bf96630f48b6b7b43dfd9645bc1d1f62145ea65a7fb682e4198 not found: ID does not exist"
Sep 17 18:16:41.690513 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:41.690486    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"]
Sep 17 18:16:41.694608 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:41.694577    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-bv6rv"]
Sep 17 18:16:42.358109 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:42.358077    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15718008-2428-4fb1-bd86-f2826369e535" path="/var/lib/kubelet/pods/15718008-2428-4fb1-bd86-f2826369e535/volumes"
Sep 17 18:16:45.128850 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:45.128810    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx" podUID="56d5664f-6361-498f-843e-ff2cbed0d58c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:16:52.385997 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:52.385965    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/2.log"
Sep 17 18:16:52.388358 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:52.388332    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/2.log"
Sep 17 18:16:55.129056 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:55.129015    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx" podUID="56d5664f-6361-498f-843e-ff2cbed0d58c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:16:55.418651 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:16:55.418560    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx" podUID="56d5664f-6361-498f-843e-ff2cbed0d58c" containerName="main" containerID="cri-o://b5c8f476733fe2134f8cde1f8568332629af0a866593c2055dd0aa655c37c437" gracePeriod=45
Sep 17 18:17:05.128991 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:05.128941    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx" podUID="56d5664f-6361-498f-843e-ff2cbed0d58c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:17:05.129473 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:05.129056    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx"
Sep 17 18:17:15.129101 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:15.129056    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx" podUID="56d5664f-6361-498f-843e-ff2cbed0d58c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:17:25.128680 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:25.128635    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx" podUID="56d5664f-6361-498f-843e-ff2cbed0d58c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:17:25.667866 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:25.667842    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx"
Sep 17 18:17:25.745930 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:25.745864    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7h6kh\" (UniqueName: \"kubernetes.io/projected/56d5664f-6361-498f-843e-ff2cbed0d58c-kube-api-access-7h6kh\") pod \"56d5664f-6361-498f-843e-ff2cbed0d58c\" (UID: \"56d5664f-6361-498f-843e-ff2cbed0d58c\") "
Sep 17 18:17:25.746135 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:25.745951    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56d5664f-6361-498f-843e-ff2cbed0d58c-tls-certs\") pod \"56d5664f-6361-498f-843e-ff2cbed0d58c\" (UID: \"56d5664f-6361-498f-843e-ff2cbed0d58c\") "
Sep 17 18:17:25.748111 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:25.748076    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56d5664f-6361-498f-843e-ff2cbed0d58c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "56d5664f-6361-498f-843e-ff2cbed0d58c" (UID: "56d5664f-6361-498f-843e-ff2cbed0d58c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:17:25.748111 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:25.748079    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d5664f-6361-498f-843e-ff2cbed0d58c-kube-api-access-7h6kh" (OuterVolumeSpecName: "kube-api-access-7h6kh") pod "56d5664f-6361-498f-843e-ff2cbed0d58c" (UID: "56d5664f-6361-498f-843e-ff2cbed0d58c"). InnerVolumeSpecName "kube-api-access-7h6kh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:17:25.846978 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:25.846935    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h6kh\" (UniqueName: \"kubernetes.io/projected/56d5664f-6361-498f-843e-ff2cbed0d58c-kube-api-access-7h6kh\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:17:25.846978 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:25.846981    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56d5664f-6361-498f-843e-ff2cbed0d58c-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:17:25.881141 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:25.881099    2581 generic.go:356] "Generic (PLEG): container finished" podID="56d5664f-6361-498f-843e-ff2cbed0d58c" containerID="b5c8f476733fe2134f8cde1f8568332629af0a866593c2055dd0aa655c37c437" exitCode=0
Sep 17 18:17:25.881297 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:25.881147    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx"
Sep 17 18:17:25.881297 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:25.881175    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx" event={"ID":"56d5664f-6361-498f-843e-ff2cbed0d58c","Type":"ContainerDied","Data":"b5c8f476733fe2134f8cde1f8568332629af0a866593c2055dd0aa655c37c437"}
Sep 17 18:17:25.881297 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:25.881227    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx" event={"ID":"56d5664f-6361-498f-843e-ff2cbed0d58c","Type":"ContainerDied","Data":"b8ee8e9d36f541a7501f1bde36db1c5123418964953e57c026ecd9a26d5ecf6b"}
Sep 17 18:17:25.881297 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:25.881256    2581 scope.go:117] "RemoveContainer" containerID="b5c8f476733fe2134f8cde1f8568332629af0a866593c2055dd0aa655c37c437"
Sep 17 18:17:25.892644 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:25.892622    2581 scope.go:117] "RemoveContainer" containerID="b5c8f476733fe2134f8cde1f8568332629af0a866593c2055dd0aa655c37c437"
Sep 17 18:17:25.892939 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:17:25.892916    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5c8f476733fe2134f8cde1f8568332629af0a866593c2055dd0aa655c37c437\": container with ID starting with b5c8f476733fe2134f8cde1f8568332629af0a866593c2055dd0aa655c37c437 not found: ID does not exist" containerID="b5c8f476733fe2134f8cde1f8568332629af0a866593c2055dd0aa655c37c437"
Sep 17 18:17:25.893015 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:25.892948    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5c8f476733fe2134f8cde1f8568332629af0a866593c2055dd0aa655c37c437"} err="failed to get container status \"b5c8f476733fe2134f8cde1f8568332629af0a866593c2055dd0aa655c37c437\": rpc error: code = NotFound desc = could not find container \"b5c8f476733fe2134f8cde1f8568332629af0a866593c2055dd0aa655c37c437\": container with ID starting with b5c8f476733fe2134f8cde1f8568332629af0a866593c2055dd0aa655c37c437 not found: ID does not exist"
Sep 17 18:17:25.916177 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:25.916133    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx"]
Sep 17 18:17:25.919770 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:25.919740    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75gd9cx"]
Sep 17 18:17:26.358445 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:26.358414    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56d5664f-6361-498f-843e-ff2cbed0d58c" path="/var/lib/kubelet/pods/56d5664f-6361-498f-843e-ff2cbed0d58c/volumes"
Sep 17 18:17:38.609648 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.609613    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"]
Sep 17 18:17:38.610029 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.610005    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15718008-2428-4fb1-bd86-f2826369e535" containerName="llm-d-routing-sidecar"
Sep 17 18:17:38.610029 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.610017    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="15718008-2428-4fb1-bd86-f2826369e535" containerName="llm-d-routing-sidecar"
Sep 17 18:17:38.610029 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.610025    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15718008-2428-4fb1-bd86-f2826369e535" containerName="main"
Sep 17 18:17:38.610029 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.610031    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="15718008-2428-4fb1-bd86-f2826369e535" containerName="main"
Sep 17 18:17:38.610251 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.610049    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56d5664f-6361-498f-843e-ff2cbed0d58c" containerName="main"
Sep 17 18:17:38.610251 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.610055    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="56d5664f-6361-498f-843e-ff2cbed0d58c" containerName="main"
Sep 17 18:17:38.610251 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.610122    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="56d5664f-6361-498f-843e-ff2cbed0d58c" containerName="main"
Sep 17 18:17:38.610251 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.610129    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="15718008-2428-4fb1-bd86-f2826369e535" containerName="llm-d-routing-sidecar"
Sep 17 18:17:38.610251 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.610137    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="15718008-2428-4fb1-bd86-f2826369e535" containerName="main"
Sep 17 18:17:38.614581 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.614561    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"
Sep 17 18:17:38.618422 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.618392    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-7ptzw\""
Sep 17 18:17:38.618422 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.618417    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 17 18:17:38.618754 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.618734    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 17 18:17:38.618817 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.618796    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-9m7c7\""
Sep 17 18:17:38.618878 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.618853    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 17 18:17:38.625223 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.625200    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"]
Sep 17 18:17:38.760985 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.760947    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/892f5168-95fb-4bc1-b3ec-81017861ffe7-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4\" (UID: \"892f5168-95fb-4bc1-b3ec-81017861ffe7\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"
Sep 17 18:17:38.761200 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.760991    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nll8z\" (UniqueName: \"kubernetes.io/projected/892f5168-95fb-4bc1-b3ec-81017861ffe7-kube-api-access-nll8z\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4\" (UID: \"892f5168-95fb-4bc1-b3ec-81017861ffe7\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"
Sep 17 18:17:38.761200 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.761130    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/892f5168-95fb-4bc1-b3ec-81017861ffe7-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4\" (UID: \"892f5168-95fb-4bc1-b3ec-81017861ffe7\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"
Sep 17 18:17:38.761200 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.761168    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/892f5168-95fb-4bc1-b3ec-81017861ffe7-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4\" (UID: \"892f5168-95fb-4bc1-b3ec-81017861ffe7\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"
Sep 17 18:17:38.761200 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.761197    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/892f5168-95fb-4bc1-b3ec-81017861ffe7-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4\" (UID: \"892f5168-95fb-4bc1-b3ec-81017861ffe7\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"
Sep 17 18:17:38.761398 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.761255    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/892f5168-95fb-4bc1-b3ec-81017861ffe7-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4\" (UID: \"892f5168-95fb-4bc1-b3ec-81017861ffe7\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"
Sep 17 18:17:38.821977 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.821936    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched64fd6"]
Sep 17 18:17:38.825365 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.825342    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched64fd6"
Sep 17 18:17:38.828161 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.828131    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-mmn7x\""
Sep 17 18:17:38.834374 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.834344    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched64fd6"]
Sep 17 18:17:38.862662 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.862570    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/892f5168-95fb-4bc1-b3ec-81017861ffe7-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4\" (UID: \"892f5168-95fb-4bc1-b3ec-81017861ffe7\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"
Sep 17 18:17:38.862662 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.862606    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/892f5168-95fb-4bc1-b3ec-81017861ffe7-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4\" (UID: \"892f5168-95fb-4bc1-b3ec-81017861ffe7\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"
Sep 17 18:17:38.862662 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.862625    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/892f5168-95fb-4bc1-b3ec-81017861ffe7-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4\" (UID: \"892f5168-95fb-4bc1-b3ec-81017861ffe7\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"
Sep 17 18:17:38.862662 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.862657    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/892f5168-95fb-4bc1-b3ec-81017861ffe7-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4\" (UID: \"892f5168-95fb-4bc1-b3ec-81017861ffe7\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"
Sep 17 18:17:38.863032 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.862678    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/892f5168-95fb-4bc1-b3ec-81017861ffe7-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4\" (UID: \"892f5168-95fb-4bc1-b3ec-81017861ffe7\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"
Sep 17 18:17:38.863032 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.862696    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nll8z\" (UniqueName: \"kubernetes.io/projected/892f5168-95fb-4bc1-b3ec-81017861ffe7-kube-api-access-nll8z\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4\" (UID: \"892f5168-95fb-4bc1-b3ec-81017861ffe7\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"
Sep 17 18:17:38.863127 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.863056    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/892f5168-95fb-4bc1-b3ec-81017861ffe7-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4\" (UID: \"892f5168-95fb-4bc1-b3ec-81017861ffe7\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"
Sep 17 18:17:38.863127 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.863106    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/892f5168-95fb-4bc1-b3ec-81017861ffe7-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4\" (UID: \"892f5168-95fb-4bc1-b3ec-81017861ffe7\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"
Sep 17 18:17:38.863216 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.863200    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/892f5168-95fb-4bc1-b3ec-81017861ffe7-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4\" (UID: \"892f5168-95fb-4bc1-b3ec-81017861ffe7\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"
Sep 17 18:17:38.865012 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.864994    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/892f5168-95fb-4bc1-b3ec-81017861ffe7-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4\" (UID: \"892f5168-95fb-4bc1-b3ec-81017861ffe7\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"
Sep 17 18:17:38.865263 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.865246    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/892f5168-95fb-4bc1-b3ec-81017861ffe7-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4\" (UID: \"892f5168-95fb-4bc1-b3ec-81017861ffe7\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"
Sep 17 18:17:38.870191 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.870172    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nll8z\" (UniqueName: \"kubernetes.io/projected/892f5168-95fb-4bc1-b3ec-81017861ffe7-kube-api-access-nll8z\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4\" (UID: \"892f5168-95fb-4bc1-b3ec-81017861ffe7\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"
Sep 17 18:17:38.926324 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.926286    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"
Sep 17 18:17:38.964235 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.964172    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcm94\" (UniqueName: \"kubernetes.io/projected/99caab94-7ff9-47a9-99ee-311aeb3c489d-kube-api-access-zcm94\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched64fd6\" (UID: \"99caab94-7ff9-47a9-99ee-311aeb3c489d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched64fd6"
Sep 17 18:17:38.964413 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:38.964316    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99caab94-7ff9-47a9-99ee-311aeb3c489d-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched64fd6\" (UID: \"99caab94-7ff9-47a9-99ee-311aeb3c489d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched64fd6"
Sep 17 18:17:39.065404 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:39.065364    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcm94\" (UniqueName: \"kubernetes.io/projected/99caab94-7ff9-47a9-99ee-311aeb3c489d-kube-api-access-zcm94\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched64fd6\" (UID: \"99caab94-7ff9-47a9-99ee-311aeb3c489d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched64fd6"
Sep 17 18:17:39.065581 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:39.065429    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99caab94-7ff9-47a9-99ee-311aeb3c489d-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched64fd6\" (UID: \"99caab94-7ff9-47a9-99ee-311aeb3c489d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched64fd6"
Sep 17 18:17:39.068098 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:39.068065    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99caab94-7ff9-47a9-99ee-311aeb3c489d-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched64fd6\" (UID: \"99caab94-7ff9-47a9-99ee-311aeb3c489d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched64fd6"
Sep 17 18:17:39.074038 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:39.074020    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcm94\" (UniqueName: \"kubernetes.io/projected/99caab94-7ff9-47a9-99ee-311aeb3c489d-kube-api-access-zcm94\") pod \"scheduler-v06-threshold-migration-test-kserve-router-sched64fd6\" (UID: \"99caab94-7ff9-47a9-99ee-311aeb3c489d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched64fd6"
Sep 17 18:17:39.136432 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:39.136333    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched64fd6"
Sep 17 18:17:39.201210 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:39.201179    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"]
Sep 17 18:17:39.202122 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:17:39.202099    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod892f5168_95fb_4bc1_b3ec_81017861ffe7.slice/crio-c34585a8d592fded8413a002de15846aff77ab6e7d890fecaaf22caed5cd6879 WatchSource:0}: Error finding container c34585a8d592fded8413a002de15846aff77ab6e7d890fecaaf22caed5cd6879: Status 404 returned error can't find the container with id c34585a8d592fded8413a002de15846aff77ab6e7d890fecaaf22caed5cd6879
Sep 17 18:17:39.409504 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:39.409477    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched64fd6"]
Sep 17 18:17:39.411873 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:17:39.411845    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99caab94_7ff9_47a9_99ee_311aeb3c489d.slice/crio-c58ae61a25e66c1876693814bee7dfafba8a5f81529161951d09967994220212 WatchSource:0}: Error finding container c58ae61a25e66c1876693814bee7dfafba8a5f81529161951d09967994220212: Status 404 returned error can't find the container with id c58ae61a25e66c1876693814bee7dfafba8a5f81529161951d09967994220212
Sep 17 18:17:39.947071 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:39.947029    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched64fd6" event={"ID":"99caab94-7ff9-47a9-99ee-311aeb3c489d","Type":"ContainerStarted","Data":"703f8de04d0205ebbd912da6949f094cc15a9fe3dc1bb8ca08369e33fc6a6355"}
Sep 17 18:17:39.947563 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:39.947081    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched64fd6" event={"ID":"99caab94-7ff9-47a9-99ee-311aeb3c489d","Type":"ContainerStarted","Data":"c58ae61a25e66c1876693814bee7dfafba8a5f81529161951d09967994220212"}
Sep 17 18:17:39.947563 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:39.947127    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched64fd6"
Sep 17 18:17:39.948757 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:39.948720    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4" event={"ID":"892f5168-95fb-4bc1-b3ec-81017861ffe7","Type":"ContainerStarted","Data":"37dd16dab6b7fef15bafeb252a22484ce3161d0e1ae606285e3efd02164f16ea"}
Sep 17 18:17:39.948896 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:39.948778    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4" event={"ID":"892f5168-95fb-4bc1-b3ec-81017861ffe7","Type":"ContainerStarted","Data":"c34585a8d592fded8413a002de15846aff77ab6e7d890fecaaf22caed5cd6879"}
Sep 17 18:17:39.948896 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:39.948810    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"
Sep 17 18:17:39.971623 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:39.971539    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched64fd6" podStartSLOduration=1.971515112 podStartE2EDuration="1.971515112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:17:39.968638656 +0000 UTC m=+1248.059260331" watchObservedRunningTime="2026-09-17 18:17:39.971515112 +0000 UTC m=+1248.062136766"
Sep 17 18:17:40.955258 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:40.955222    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4" event={"ID":"892f5168-95fb-4bc1-b3ec-81017861ffe7","Type":"ContainerStarted","Data":"68a074bf9c62c262627355f8289c3dd7cebc3e0de730c09ce1e1c90c99027afe"}
Sep 17 18:17:40.981945 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:40.981872    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4" podStartSLOduration=2.981855914 podStartE2EDuration="2.981855914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:17:40.980513133 +0000 UTC m=+1249.071134787" watchObservedRunningTime="2026-09-17 18:17:40.981855914 +0000 UTC m=+1249.072477578"
Sep 17 18:17:41.926616 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:41.926581    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"
Sep 17 18:17:48.927134 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:48.927038    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"
Sep 17 18:17:48.939511 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:48.939484    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"
Sep 17 18:17:49.004856 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:49.004828    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"
Sep 17 18:17:50.011166 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:17:50.011139    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"
Sep 17 18:18:10.957428 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:10.957398    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched64fd6"
Sep 17 18:18:16.219834 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:16.219793    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched64fd6"]
Sep 17 18:18:16.229373 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:16.229341    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"]
Sep 17 18:18:16.239573 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:18:16.239525    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:18:16.239573 ip-10-0-130-28 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:18:16.239573 ip-10-0-130-28 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:18:16.239573 ip-10-0-130-28 kubenswrapper[2581]:  > containerID="68a074bf9c62c262627355f8289c3dd7cebc3e0de730c09ce1e1c90c99027afe" cmd=["/bin/sleep","15"]
Sep 17 18:18:16.239929 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:16.239601    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4" message="" err=<
Sep 17 18:18:16.239929 ip-10-0-130-28 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:18:16.239929 ip-10-0-130-28 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:18:16.239929 ip-10-0-130-28 kubenswrapper[2581]:  >
Sep 17 18:18:16.239929 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:18:16.239620    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:18:16.239929 ip-10-0-130-28 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:18:16.239929 ip-10-0-130-28 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:18:16.239929 ip-10-0-130-28 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4" podUID="892f5168-95fb-4bc1-b3ec-81017861ffe7" containerName="main" containerID="cri-o://68a074bf9c62c262627355f8289c3dd7cebc3e0de730c09ce1e1c90c99027afe"
Sep 17 18:18:16.239929 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:16.239688    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4" podUID="892f5168-95fb-4bc1-b3ec-81017861ffe7" containerName="main" containerID="cri-o://68a074bf9c62c262627355f8289c3dd7cebc3e0de730c09ce1e1c90c99027afe" gracePeriod=60
Sep 17 18:18:16.455767 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:16.455735    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4" podUID="892f5168-95fb-4bc1-b3ec-81017861ffe7" containerName="llm-d-routing-sidecar" containerID="cri-o://37dd16dab6b7fef15bafeb252a22484ce3161d0e1ae606285e3efd02164f16ea" gracePeriod=60
Sep 17 18:18:16.706324 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:16.706293    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"
Sep 17 18:18:16.802021 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:16.801926    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/892f5168-95fb-4bc1-b3ec-81017861ffe7-tmp-dir\") pod \"892f5168-95fb-4bc1-b3ec-81017861ffe7\" (UID: \"892f5168-95fb-4bc1-b3ec-81017861ffe7\") "
Sep 17 18:18:16.802021 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:16.801964    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/892f5168-95fb-4bc1-b3ec-81017861ffe7-tls-certs\") pod \"892f5168-95fb-4bc1-b3ec-81017861ffe7\" (UID: \"892f5168-95fb-4bc1-b3ec-81017861ffe7\") "
Sep 17 18:18:16.802021 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:16.801996    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/892f5168-95fb-4bc1-b3ec-81017861ffe7-model-cache\") pod \"892f5168-95fb-4bc1-b3ec-81017861ffe7\" (UID: \"892f5168-95fb-4bc1-b3ec-81017861ffe7\") "
Sep 17 18:18:16.802307 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:16.802043    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nll8z\" (UniqueName: \"kubernetes.io/projected/892f5168-95fb-4bc1-b3ec-81017861ffe7-kube-api-access-nll8z\") pod \"892f5168-95fb-4bc1-b3ec-81017861ffe7\" (UID: \"892f5168-95fb-4bc1-b3ec-81017861ffe7\") "
Sep 17 18:18:16.802307 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:16.802068    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/892f5168-95fb-4bc1-b3ec-81017861ffe7-dshm\") pod \"892f5168-95fb-4bc1-b3ec-81017861ffe7\" (UID: \"892f5168-95fb-4bc1-b3ec-81017861ffe7\") "
Sep 17 18:18:16.802307 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:16.802091    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/892f5168-95fb-4bc1-b3ec-81017861ffe7-home\") pod \"892f5168-95fb-4bc1-b3ec-81017861ffe7\" (UID: \"892f5168-95fb-4bc1-b3ec-81017861ffe7\") "
Sep 17 18:18:16.802307 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:16.802237    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892f5168-95fb-4bc1-b3ec-81017861ffe7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "892f5168-95fb-4bc1-b3ec-81017861ffe7" (UID: "892f5168-95fb-4bc1-b3ec-81017861ffe7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:16.802497 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:16.802331    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892f5168-95fb-4bc1-b3ec-81017861ffe7-home" (OuterVolumeSpecName: "home") pod "892f5168-95fb-4bc1-b3ec-81017861ffe7" (UID: "892f5168-95fb-4bc1-b3ec-81017861ffe7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:16.802497 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:16.802342    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892f5168-95fb-4bc1-b3ec-81017861ffe7-model-cache" (OuterVolumeSpecName: "model-cache") pod "892f5168-95fb-4bc1-b3ec-81017861ffe7" (UID: "892f5168-95fb-4bc1-b3ec-81017861ffe7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:16.802497 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:16.802452    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/892f5168-95fb-4bc1-b3ec-81017861ffe7-model-cache\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:18:16.802497 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:16.802473    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/892f5168-95fb-4bc1-b3ec-81017861ffe7-home\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:18:16.802497 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:16.802486    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/892f5168-95fb-4bc1-b3ec-81017861ffe7-tmp-dir\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:18:16.804339 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:16.804317    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/892f5168-95fb-4bc1-b3ec-81017861ffe7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "892f5168-95fb-4bc1-b3ec-81017861ffe7" (UID: "892f5168-95fb-4bc1-b3ec-81017861ffe7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:18:16.804407 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:16.804330    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892f5168-95fb-4bc1-b3ec-81017861ffe7-dshm" (OuterVolumeSpecName: "dshm") pod "892f5168-95fb-4bc1-b3ec-81017861ffe7" (UID: "892f5168-95fb-4bc1-b3ec-81017861ffe7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:16.804407 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:16.804323    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892f5168-95fb-4bc1-b3ec-81017861ffe7-kube-api-access-nll8z" (OuterVolumeSpecName: "kube-api-access-nll8z") pod "892f5168-95fb-4bc1-b3ec-81017861ffe7" (UID: "892f5168-95fb-4bc1-b3ec-81017861ffe7"). InnerVolumeSpecName "kube-api-access-nll8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:18:16.903297 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:16.903263    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/892f5168-95fb-4bc1-b3ec-81017861ffe7-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:18:16.903297 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:16.903294    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nll8z\" (UniqueName: \"kubernetes.io/projected/892f5168-95fb-4bc1-b3ec-81017861ffe7-kube-api-access-nll8z\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:18:16.903297 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:16.903304    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/892f5168-95fb-4bc1-b3ec-81017861ffe7-dshm\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:18:17.128654 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:17.128558    2581 generic.go:356] "Generic (PLEG): container finished" podID="892f5168-95fb-4bc1-b3ec-81017861ffe7" containerID="68a074bf9c62c262627355f8289c3dd7cebc3e0de730c09ce1e1c90c99027afe" exitCode=0
Sep 17 18:18:17.128654 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:17.128594    2581 generic.go:356] "Generic (PLEG): container finished" podID="892f5168-95fb-4bc1-b3ec-81017861ffe7" containerID="37dd16dab6b7fef15bafeb252a22484ce3161d0e1ae606285e3efd02164f16ea" exitCode=0
Sep 17 18:18:17.128654 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:17.128617    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"
Sep 17 18:18:17.128654 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:17.128629    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4" event={"ID":"892f5168-95fb-4bc1-b3ec-81017861ffe7","Type":"ContainerDied","Data":"68a074bf9c62c262627355f8289c3dd7cebc3e0de730c09ce1e1c90c99027afe"}
Sep 17 18:18:17.128951 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:17.128685    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4" event={"ID":"892f5168-95fb-4bc1-b3ec-81017861ffe7","Type":"ContainerDied","Data":"37dd16dab6b7fef15bafeb252a22484ce3161d0e1ae606285e3efd02164f16ea"}
Sep 17 18:18:17.128951 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:17.128710    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4" event={"ID":"892f5168-95fb-4bc1-b3ec-81017861ffe7","Type":"ContainerDied","Data":"c34585a8d592fded8413a002de15846aff77ab6e7d890fecaaf22caed5cd6879"}
Sep 17 18:18:17.128951 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:17.128740    2581 scope.go:117] "RemoveContainer" containerID="68a074bf9c62c262627355f8289c3dd7cebc3e0de730c09ce1e1c90c99027afe"
Sep 17 18:18:17.137816 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:17.137797    2581 scope.go:117] "RemoveContainer" containerID="37dd16dab6b7fef15bafeb252a22484ce3161d0e1ae606285e3efd02164f16ea"
Sep 17 18:18:17.145968 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:17.145944    2581 scope.go:117] "RemoveContainer" containerID="68a074bf9c62c262627355f8289c3dd7cebc3e0de730c09ce1e1c90c99027afe"
Sep 17 18:18:17.146245 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:18:17.146225    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68a074bf9c62c262627355f8289c3dd7cebc3e0de730c09ce1e1c90c99027afe\": container with ID starting with 68a074bf9c62c262627355f8289c3dd7cebc3e0de730c09ce1e1c90c99027afe not found: ID does not exist" containerID="68a074bf9c62c262627355f8289c3dd7cebc3e0de730c09ce1e1c90c99027afe"
Sep 17 18:18:17.146317 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:17.146252    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68a074bf9c62c262627355f8289c3dd7cebc3e0de730c09ce1e1c90c99027afe"} err="failed to get container status \"68a074bf9c62c262627355f8289c3dd7cebc3e0de730c09ce1e1c90c99027afe\": rpc error: code = NotFound desc = could not find container \"68a074bf9c62c262627355f8289c3dd7cebc3e0de730c09ce1e1c90c99027afe\": container with ID starting with 68a074bf9c62c262627355f8289c3dd7cebc3e0de730c09ce1e1c90c99027afe not found: ID does not exist"
Sep 17 18:18:17.146317 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:17.146270    2581 scope.go:117] "RemoveContainer" containerID="37dd16dab6b7fef15bafeb252a22484ce3161d0e1ae606285e3efd02164f16ea"
Sep 17 18:18:17.146486 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:18:17.146469    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37dd16dab6b7fef15bafeb252a22484ce3161d0e1ae606285e3efd02164f16ea\": container with ID starting with 37dd16dab6b7fef15bafeb252a22484ce3161d0e1ae606285e3efd02164f16ea not found: ID does not exist" containerID="37dd16dab6b7fef15bafeb252a22484ce3161d0e1ae606285e3efd02164f16ea"
Sep 17 18:18:17.146524 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:17.146492    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37dd16dab6b7fef15bafeb252a22484ce3161d0e1ae606285e3efd02164f16ea"} err="failed to get container status \"37dd16dab6b7fef15bafeb252a22484ce3161d0e1ae606285e3efd02164f16ea\": rpc error: code = NotFound desc = could not find container \"37dd16dab6b7fef15bafeb252a22484ce3161d0e1ae606285e3efd02164f16ea\": container with ID starting with 37dd16dab6b7fef15bafeb252a22484ce3161d0e1ae606285e3efd02164f16ea not found: ID does not exist"
Sep 17 18:18:17.146524 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:17.146506    2581 scope.go:117] "RemoveContainer" containerID="68a074bf9c62c262627355f8289c3dd7cebc3e0de730c09ce1e1c90c99027afe"
Sep 17 18:18:17.146709 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:17.146693    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68a074bf9c62c262627355f8289c3dd7cebc3e0de730c09ce1e1c90c99027afe"} err="failed to get container status \"68a074bf9c62c262627355f8289c3dd7cebc3e0de730c09ce1e1c90c99027afe\": rpc error: code = NotFound desc = could not find container \"68a074bf9c62c262627355f8289c3dd7cebc3e0de730c09ce1e1c90c99027afe\": container with ID starting with 68a074bf9c62c262627355f8289c3dd7cebc3e0de730c09ce1e1c90c99027afe not found: ID does not exist"
Sep 17 18:18:17.146753 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:17.146708    2581 scope.go:117] "RemoveContainer" containerID="37dd16dab6b7fef15bafeb252a22484ce3161d0e1ae606285e3efd02164f16ea"
Sep 17 18:18:17.146971 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:17.146955    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37dd16dab6b7fef15bafeb252a22484ce3161d0e1ae606285e3efd02164f16ea"} err="failed to get container status \"37dd16dab6b7fef15bafeb252a22484ce3161d0e1ae606285e3efd02164f16ea\": rpc error: code = NotFound desc = could not find container \"37dd16dab6b7fef15bafeb252a22484ce3161d0e1ae606285e3efd02164f16ea\": container with ID starting with 37dd16dab6b7fef15bafeb252a22484ce3161d0e1ae606285e3efd02164f16ea not found: ID does not exist"
Sep 17 18:18:17.158047 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:17.158009    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"]
Sep 17 18:18:17.158272 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:17.158069    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-9tvd4"]
Sep 17 18:18:18.358812 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:18.358781    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="892f5168-95fb-4bc1-b3ec-81017861ffe7" path="/var/lib/kubelet/pods/892f5168-95fb-4bc1-b3ec-81017861ffe7/volumes"
Sep 17 18:18:20.957434 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:20.957397    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched64fd6" podUID="99caab94-7ff9-47a9-99ee-311aeb3c489d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:18:30.956818 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:30.956780    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched64fd6" podUID="99caab94-7ff9-47a9-99ee-311aeb3c489d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:18:31.241922 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:31.241816    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched64fd6" podUID="99caab94-7ff9-47a9-99ee-311aeb3c489d" containerName="main" containerID="cri-o://703f8de04d0205ebbd912da6949f094cc15a9fe3dc1bb8ca08369e33fc6a6355" gracePeriod=45
Sep 17 18:18:37.922242 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:37.922199    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm"]
Sep 17 18:18:40.957324 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:40.957287    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched64fd6" podUID="99caab94-7ff9-47a9-99ee-311aeb3c489d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:18:40.957747 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:40.957382    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched64fd6"
Sep 17 18:18:45.495090 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:45.495051    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm" podUID="b6bc9e69-b6b3-4c16-be12-853b6f9b5764" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:18:50.956697 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:50.956657    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched64fd6" podUID="99caab94-7ff9-47a9-99ee-311aeb3c489d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:18:52.942847 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:52.942742    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm" podUID="b6bc9e69-b6b3-4c16-be12-853b6f9b5764" containerName="main" containerID="cri-o://527783aca1b7667f00ecdec4ccac254717a9c9bd28447fa8805b1df24ba2c319" gracePeriod=45
Sep 17 18:18:55.495567 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:18:55.495528    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm" podUID="b6bc9e69-b6b3-4c16-be12-853b6f9b5764" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:19:00.957248 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:00.957208    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched64fd6" podUID="99caab94-7ff9-47a9-99ee-311aeb3c489d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:19:01.488968 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:01.488943    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched64fd6"
Sep 17 18:19:01.608457 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:01.608368    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99caab94-7ff9-47a9-99ee-311aeb3c489d-tls-certs\") pod \"99caab94-7ff9-47a9-99ee-311aeb3c489d\" (UID: \"99caab94-7ff9-47a9-99ee-311aeb3c489d\") "
Sep 17 18:19:01.608457 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:01.608449    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcm94\" (UniqueName: \"kubernetes.io/projected/99caab94-7ff9-47a9-99ee-311aeb3c489d-kube-api-access-zcm94\") pod \"99caab94-7ff9-47a9-99ee-311aeb3c489d\" (UID: \"99caab94-7ff9-47a9-99ee-311aeb3c489d\") "
Sep 17 18:19:01.610565 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:01.610528    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99caab94-7ff9-47a9-99ee-311aeb3c489d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "99caab94-7ff9-47a9-99ee-311aeb3c489d" (UID: "99caab94-7ff9-47a9-99ee-311aeb3c489d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:19:01.610565 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:01.610528    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99caab94-7ff9-47a9-99ee-311aeb3c489d-kube-api-access-zcm94" (OuterVolumeSpecName: "kube-api-access-zcm94") pod "99caab94-7ff9-47a9-99ee-311aeb3c489d" (UID: "99caab94-7ff9-47a9-99ee-311aeb3c489d"). InnerVolumeSpecName "kube-api-access-zcm94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:19:01.709455 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:01.709419    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99caab94-7ff9-47a9-99ee-311aeb3c489d-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:19:01.709455 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:01.709452    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcm94\" (UniqueName: \"kubernetes.io/projected/99caab94-7ff9-47a9-99ee-311aeb3c489d-kube-api-access-zcm94\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:19:02.345057 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:02.345020    2581 generic.go:356] "Generic (PLEG): container finished" podID="99caab94-7ff9-47a9-99ee-311aeb3c489d" containerID="703f8de04d0205ebbd912da6949f094cc15a9fe3dc1bb8ca08369e33fc6a6355" exitCode=0
Sep 17 18:19:02.345455 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:02.345071    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched64fd6"
Sep 17 18:19:02.345455 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:02.345085    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched64fd6" event={"ID":"99caab94-7ff9-47a9-99ee-311aeb3c489d","Type":"ContainerDied","Data":"703f8de04d0205ebbd912da6949f094cc15a9fe3dc1bb8ca08369e33fc6a6355"}
Sep 17 18:19:02.345455 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:02.345135    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched64fd6" event={"ID":"99caab94-7ff9-47a9-99ee-311aeb3c489d","Type":"ContainerDied","Data":"c58ae61a25e66c1876693814bee7dfafba8a5f81529161951d09967994220212"}
Sep 17 18:19:02.345455 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:02.345166    2581 scope.go:117] "RemoveContainer" containerID="703f8de04d0205ebbd912da6949f094cc15a9fe3dc1bb8ca08369e33fc6a6355"
Sep 17 18:19:02.354221 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:02.354203    2581 scope.go:117] "RemoveContainer" containerID="703f8de04d0205ebbd912da6949f094cc15a9fe3dc1bb8ca08369e33fc6a6355"
Sep 17 18:19:02.354488 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:19:02.354460    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"703f8de04d0205ebbd912da6949f094cc15a9fe3dc1bb8ca08369e33fc6a6355\": container with ID starting with 703f8de04d0205ebbd912da6949f094cc15a9fe3dc1bb8ca08369e33fc6a6355 not found: ID does not exist" containerID="703f8de04d0205ebbd912da6949f094cc15a9fe3dc1bb8ca08369e33fc6a6355"
Sep 17 18:19:02.354551 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:02.354498    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"703f8de04d0205ebbd912da6949f094cc15a9fe3dc1bb8ca08369e33fc6a6355"} err="failed to get container status \"703f8de04d0205ebbd912da6949f094cc15a9fe3dc1bb8ca08369e33fc6a6355\": rpc error: code = NotFound desc = could not find container \"703f8de04d0205ebbd912da6949f094cc15a9fe3dc1bb8ca08369e33fc6a6355\": container with ID starting with 703f8de04d0205ebbd912da6949f094cc15a9fe3dc1bb8ca08369e33fc6a6355 not found: ID does not exist"
Sep 17 18:19:02.368142 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:02.368113    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched64fd6"]
Sep 17 18:19:02.371693 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:02.371670    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-sched64fd6"]
Sep 17 18:19:04.358131 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:04.358099    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99caab94-7ff9-47a9-99ee-311aeb3c489d" path="/var/lib/kubelet/pods/99caab94-7ff9-47a9-99ee-311aeb3c489d/volumes"
Sep 17 18:19:05.495425 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:05.495390    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm" podUID="b6bc9e69-b6b3-4c16-be12-853b6f9b5764" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:19:05.495792 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:05.495484    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm"
Sep 17 18:19:15.495819 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:15.495728    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm" podUID="b6bc9e69-b6b3-4c16-be12-853b6f9b5764" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:19:21.271261 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.271230    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"]
Sep 17 18:19:21.271746 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.271572    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="892f5168-95fb-4bc1-b3ec-81017861ffe7" containerName="llm-d-routing-sidecar"
Sep 17 18:19:21.271746 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.271584    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="892f5168-95fb-4bc1-b3ec-81017861ffe7" containerName="llm-d-routing-sidecar"
Sep 17 18:19:21.271746 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.271596    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99caab94-7ff9-47a9-99ee-311aeb3c489d" containerName="main"
Sep 17 18:19:21.271746 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.271602    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="99caab94-7ff9-47a9-99ee-311aeb3c489d" containerName="main"
Sep 17 18:19:21.271746 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.271617    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="892f5168-95fb-4bc1-b3ec-81017861ffe7" containerName="main"
Sep 17 18:19:21.271746 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.271622    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="892f5168-95fb-4bc1-b3ec-81017861ffe7" containerName="main"
Sep 17 18:19:21.271746 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.271679    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="99caab94-7ff9-47a9-99ee-311aeb3c489d" containerName="main"
Sep 17 18:19:21.271746 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.271688    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="892f5168-95fb-4bc1-b3ec-81017861ffe7" containerName="llm-d-routing-sidecar"
Sep 17 18:19:21.271746 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.271694    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="892f5168-95fb-4bc1-b3ec-81017861ffe7" containerName="main"
Sep 17 18:19:21.276454 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.276434    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"
Sep 17 18:19:21.278813 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.278789    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 17 18:19:21.279345 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.279326    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 17 18:19:21.280269 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.280249    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:19:21.280370 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.280249    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-nkvkf\""
Sep 17 18:19:21.280370 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.280332    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 17 18:19:21.280527 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.280500    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 17 18:19:21.283982 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.283661    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"]
Sep 17 18:19:21.376137 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.376099    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fc343e6-bb26-4c06-a6c4-daa249f05175-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl\" (UID: \"8fc343e6-bb26-4c06-a6c4-daa249f05175\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"
Sep 17 18:19:21.376335 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.376164    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc343e6-bb26-4c06-a6c4-daa249f05175-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl\" (UID: \"8fc343e6-bb26-4c06-a6c4-daa249f05175\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"
Sep 17 18:19:21.376335 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.376192    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fc343e6-bb26-4c06-a6c4-daa249f05175-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl\" (UID: \"8fc343e6-bb26-4c06-a6c4-daa249f05175\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"
Sep 17 18:19:21.376335 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.376218    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc343e6-bb26-4c06-a6c4-daa249f05175-dshm\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl\" (UID: \"8fc343e6-bb26-4c06-a6c4-daa249f05175\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"
Sep 17 18:19:21.376335 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.376245    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc343e6-bb26-4c06-a6c4-daa249f05175-home\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl\" (UID: \"8fc343e6-bb26-4c06-a6c4-daa249f05175\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"
Sep 17 18:19:21.376335 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.376294    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc343e6-bb26-4c06-a6c4-daa249f05175-model-cache\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl\" (UID: \"8fc343e6-bb26-4c06-a6c4-daa249f05175\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"
Sep 17 18:19:21.376335 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.376323    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrr88\" (UniqueName: \"kubernetes.io/projected/8fc343e6-bb26-4c06-a6c4-daa249f05175-kube-api-access-qrr88\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl\" (UID: \"8fc343e6-bb26-4c06-a6c4-daa249f05175\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"
Sep 17 18:19:21.376554 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.376345    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc343e6-bb26-4c06-a6c4-daa249f05175-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl\" (UID: \"8fc343e6-bb26-4c06-a6c4-daa249f05175\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"
Sep 17 18:19:21.477432 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.477403    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc343e6-bb26-4c06-a6c4-daa249f05175-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl\" (UID: \"8fc343e6-bb26-4c06-a6c4-daa249f05175\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"
Sep 17 18:19:21.477619 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.477479    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fc343e6-bb26-4c06-a6c4-daa249f05175-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl\" (UID: \"8fc343e6-bb26-4c06-a6c4-daa249f05175\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"
Sep 17 18:19:21.477619 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.477515    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc343e6-bb26-4c06-a6c4-daa249f05175-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl\" (UID: \"8fc343e6-bb26-4c06-a6c4-daa249f05175\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"
Sep 17 18:19:21.477619 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.477538    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fc343e6-bb26-4c06-a6c4-daa249f05175-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl\" (UID: \"8fc343e6-bb26-4c06-a6c4-daa249f05175\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"
Sep 17 18:19:21.477619 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.477562    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc343e6-bb26-4c06-a6c4-daa249f05175-dshm\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl\" (UID: \"8fc343e6-bb26-4c06-a6c4-daa249f05175\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"
Sep 17 18:19:21.477619 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.477585    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc343e6-bb26-4c06-a6c4-daa249f05175-home\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl\" (UID: \"8fc343e6-bb26-4c06-a6c4-daa249f05175\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"
Sep 17 18:19:21.477941 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.477623    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc343e6-bb26-4c06-a6c4-daa249f05175-model-cache\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl\" (UID: \"8fc343e6-bb26-4c06-a6c4-daa249f05175\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"
Sep 17 18:19:21.477941 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.477653    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrr88\" (UniqueName: \"kubernetes.io/projected/8fc343e6-bb26-4c06-a6c4-daa249f05175-kube-api-access-qrr88\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl\" (UID: \"8fc343e6-bb26-4c06-a6c4-daa249f05175\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"
Sep 17 18:19:21.477941 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.477811    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc343e6-bb26-4c06-a6c4-daa249f05175-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl\" (UID: \"8fc343e6-bb26-4c06-a6c4-daa249f05175\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"
Sep 17 18:19:21.478117 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.478013    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fc343e6-bb26-4c06-a6c4-daa249f05175-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl\" (UID: \"8fc343e6-bb26-4c06-a6c4-daa249f05175\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"
Sep 17 18:19:21.478176 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.478131    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc343e6-bb26-4c06-a6c4-daa249f05175-model-cache\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl\" (UID: \"8fc343e6-bb26-4c06-a6c4-daa249f05175\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"
Sep 17 18:19:21.478316 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.478295    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fc343e6-bb26-4c06-a6c4-daa249f05175-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl\" (UID: \"8fc343e6-bb26-4c06-a6c4-daa249f05175\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"
Sep 17 18:19:21.478588 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.478564    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc343e6-bb26-4c06-a6c4-daa249f05175-home\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl\" (UID: \"8fc343e6-bb26-4c06-a6c4-daa249f05175\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"
Sep 17 18:19:21.480386 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.480358    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc343e6-bb26-4c06-a6c4-daa249f05175-dshm\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl\" (UID: \"8fc343e6-bb26-4c06-a6c4-daa249f05175\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"
Sep 17 18:19:21.480502 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.480452    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc343e6-bb26-4c06-a6c4-daa249f05175-tls-certs\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl\" (UID: \"8fc343e6-bb26-4c06-a6c4-daa249f05175\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"
Sep 17 18:19:21.492831 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.492804    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m"]
Sep 17 18:19:21.493361 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.493338    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrr88\" (UniqueName: \"kubernetes.io/projected/8fc343e6-bb26-4c06-a6c4-daa249f05175-kube-api-access-qrr88\") pod \"tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl\" (UID: \"8fc343e6-bb26-4c06-a6c4-daa249f05175\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"
Sep 17 18:19:21.497882 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.497860    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m"
Sep 17 18:19:21.500386 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.500356    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-fklrl\""
Sep 17 18:19:21.505544 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.505521    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m"]
Sep 17 18:19:21.578714 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.578617    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7nth\" (UniqueName: \"kubernetes.io/projected/0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb-kube-api-access-w7nth\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m\" (UID: \"0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m"
Sep 17 18:19:21.578714 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.578684    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m\" (UID: \"0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m"
Sep 17 18:19:21.588116 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.588090    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"
Sep 17 18:19:21.679564 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.679529    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7nth\" (UniqueName: \"kubernetes.io/projected/0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb-kube-api-access-w7nth\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m\" (UID: \"0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m"
Sep 17 18:19:21.679721 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.679601    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m\" (UID: \"0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m"
Sep 17 18:19:21.682059 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.682033    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m\" (UID: \"0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m"
Sep 17 18:19:21.687796 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.687776    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7nth\" (UniqueName: \"kubernetes.io/projected/0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb-kube-api-access-w7nth\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m\" (UID: \"0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m"
Sep 17 18:19:21.815035 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.814990    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m"
Sep 17 18:19:21.884070 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:21.884041    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"]
Sep 17 18:19:21.885712 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:19:21.885685    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fc343e6_bb26_4c06_a6c4_daa249f05175.slice/crio-2d83606114109791357ecd2c28eb279ee6b0dc1d6578dc1b6eb16ddca823178b WatchSource:0}: Error finding container 2d83606114109791357ecd2c28eb279ee6b0dc1d6578dc1b6eb16ddca823178b: Status 404 returned error can't find the container with id 2d83606114109791357ecd2c28eb279ee6b0dc1d6578dc1b6eb16ddca823178b
Sep 17 18:19:22.099103 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:22.099078    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m"]
Sep 17 18:19:22.436825 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:22.436727    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m" event={"ID":"0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb","Type":"ContainerStarted","Data":"70d4d50bce2f75e2afdf5191004ba0e37b799196f5f2790194cb069520ab16bb"}
Sep 17 18:19:22.436825 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:22.436787    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m" event={"ID":"0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb","Type":"ContainerStarted","Data":"ab20cf924b6654a8a26050af90daac9107b19d922f86e07bce18a88082c1f658"}
Sep 17 18:19:22.437380 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:22.436827    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m"
Sep 17 18:19:22.438050 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:22.438020    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl" event={"ID":"8fc343e6-bb26-4c06-a6c4-daa249f05175","Type":"ContainerStarted","Data":"2d83606114109791357ecd2c28eb279ee6b0dc1d6578dc1b6eb16ddca823178b"}
Sep 17 18:19:22.454321 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:22.454267    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m" podStartSLOduration=1.454252165 podStartE2EDuration="1.454252165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:19:22.453086842 +0000 UTC m=+1350.543708494" watchObservedRunningTime="2026-09-17 18:19:22.454252165 +0000 UTC m=+1350.544873835"
Sep 17 18:19:23.445775 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:23.445690    2581 generic.go:356] "Generic (PLEG): container finished" podID="b6bc9e69-b6b3-4c16-be12-853b6f9b5764" containerID="527783aca1b7667f00ecdec4ccac254717a9c9bd28447fa8805b1df24ba2c319" exitCode=0
Sep 17 18:19:23.446199 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:23.445771    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm" event={"ID":"b6bc9e69-b6b3-4c16-be12-853b6f9b5764","Type":"ContainerDied","Data":"527783aca1b7667f00ecdec4ccac254717a9c9bd28447fa8805b1df24ba2c319"}
Sep 17 18:19:24.026661 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:24.026633    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm"
Sep 17 18:19:24.103829 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:24.103793    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6bc9e69-b6b3-4c16-be12-853b6f9b5764-tls-certs\") pod \"b6bc9e69-b6b3-4c16-be12-853b6f9b5764\" (UID: \"b6bc9e69-b6b3-4c16-be12-853b6f9b5764\") "
Sep 17 18:19:24.103829 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:24.104005    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gz5xq\" (UniqueName: \"kubernetes.io/projected/b6bc9e69-b6b3-4c16-be12-853b6f9b5764-kube-api-access-gz5xq\") pod \"b6bc9e69-b6b3-4c16-be12-853b6f9b5764\" (UID: \"b6bc9e69-b6b3-4c16-be12-853b6f9b5764\") "
Sep 17 18:19:24.108521 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:24.108221    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6bc9e69-b6b3-4c16-be12-853b6f9b5764-kube-api-access-gz5xq" (OuterVolumeSpecName: "kube-api-access-gz5xq") pod "b6bc9e69-b6b3-4c16-be12-853b6f9b5764" (UID: "b6bc9e69-b6b3-4c16-be12-853b6f9b5764"). InnerVolumeSpecName "kube-api-access-gz5xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:19:24.108658 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:24.108339    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6bc9e69-b6b3-4c16-be12-853b6f9b5764-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b6bc9e69-b6b3-4c16-be12-853b6f9b5764" (UID: "b6bc9e69-b6b3-4c16-be12-853b6f9b5764"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:19:24.205460 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:24.205279    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz5xq\" (UniqueName: \"kubernetes.io/projected/b6bc9e69-b6b3-4c16-be12-853b6f9b5764-kube-api-access-gz5xq\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:19:24.205678 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:24.205473    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6bc9e69-b6b3-4c16-be12-853b6f9b5764-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:19:24.453960 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:24.453925    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm"
Sep 17 18:19:24.453960 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:24.453942    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm" event={"ID":"b6bc9e69-b6b3-4c16-be12-853b6f9b5764","Type":"ContainerDied","Data":"5614cdca5089073aedbf10530c3bc9a6d5c836fa4bbb614746d8de3e3e302398"}
Sep 17 18:19:24.454439 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:24.454023    2581 scope.go:117] "RemoveContainer" containerID="527783aca1b7667f00ecdec4ccac254717a9c9bd28447fa8805b1df24ba2c319"
Sep 17 18:19:24.472766 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:24.472736    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm"]
Sep 17 18:19:24.477652 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:24.477627    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-x9gvm"]
Sep 17 18:19:26.363557 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:26.363512    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6bc9e69-b6b3-4c16-be12-853b6f9b5764" path="/var/lib/kubelet/pods/b6bc9e69-b6b3-4c16-be12-853b6f9b5764/volumes"
Sep 17 18:19:26.468409 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:26.468370    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl" event={"ID":"8fc343e6-bb26-4c06-a6c4-daa249f05175","Type":"ContainerStarted","Data":"650f1f58809d46c9a7ad0aecc4bb9bcfc6e02630ea70f05cac98245f1311bd62"}
Sep 17 18:19:28.478286 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:28.478257    2581 generic.go:356] "Generic (PLEG): container finished" podID="8fc343e6-bb26-4c06-a6c4-daa249f05175" containerID="650f1f58809d46c9a7ad0aecc4bb9bcfc6e02630ea70f05cac98245f1311bd62" exitCode=0
Sep 17 18:19:28.478661 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:28.478328    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl" event={"ID":"8fc343e6-bb26-4c06-a6c4-daa249f05175","Type":"ContainerDied","Data":"650f1f58809d46c9a7ad0aecc4bb9bcfc6e02630ea70f05cac98245f1311bd62"}
Sep 17 18:19:29.485328 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:29.485292    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl" event={"ID":"8fc343e6-bb26-4c06-a6c4-daa249f05175","Type":"ContainerStarted","Data":"3fffabdfb494f25fd9452f3cf05f2174466ea9ea282ce4a1491b48d40540c32d"}
Sep 17 18:19:29.506501 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:29.506448    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl" podStartSLOduration=2.3054873049999998 podStartE2EDuration="8.506429482s" totalImagesPullingTime="4.025270042s" totalInitContainerRuntime="2.175672135s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:19:29.505597061 +0000 UTC m=+1357.596218748" watchObservedRunningTime="2026-09-17 18:19:29.506429482 +0000 UTC m=+1357.597051181"
Sep 17 18:19:29.589137 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:29.589101    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"
Sep 17 18:19:31.588986 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:31.588944    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"
Sep 17 18:19:31.601291 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:31.601267    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"
Sep 17 18:19:32.510754 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:32.510728    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"
Sep 17 18:19:53.448297 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:19:53.448258    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m"
Sep 17 18:20:06.881666 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:06.881626    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"]
Sep 17 18:20:06.882095 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:06.882080    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6bc9e69-b6b3-4c16-be12-853b6f9b5764" containerName="main"
Sep 17 18:20:06.882140 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:06.882096    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6bc9e69-b6b3-4c16-be12-853b6f9b5764" containerName="main"
Sep 17 18:20:06.882182 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:06.882164    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6bc9e69-b6b3-4c16-be12-853b6f9b5764" containerName="main"
Sep 17 18:20:06.886686 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:06.886664    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"
Sep 17 18:20:06.889479 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:06.889454    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 17 18:20:06.889605 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:06.889482    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:20:06.889605 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:06.889454    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 17 18:20:06.889605 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:06.889530    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 17 18:20:06.889916 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:06.889882    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Sep 17 18:20:06.890015 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:06.889966    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-fb8mf\""
Sep 17 18:20:06.896951 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:06.896926    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"]
Sep 17 18:20:07.005845 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.005807    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5caf49c5-c902-4acd-949b-753c244a55c9-dshm\") pod \"auth-invalid-token-test-kserve-849f576695-69lfl\" (UID: \"5caf49c5-c902-4acd-949b-753c244a55c9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"
Sep 17 18:20:07.005845 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.005847    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5caf49c5-c902-4acd-949b-753c244a55c9-model-cache\") pod \"auth-invalid-token-test-kserve-849f576695-69lfl\" (UID: \"5caf49c5-c902-4acd-949b-753c244a55c9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"
Sep 17 18:20:07.006080 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.005885    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5caf49c5-c902-4acd-949b-753c244a55c9-tls-certs\") pod \"auth-invalid-token-test-kserve-849f576695-69lfl\" (UID: \"5caf49c5-c902-4acd-949b-753c244a55c9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"
Sep 17 18:20:07.006080 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.005945    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5caf49c5-c902-4acd-949b-753c244a55c9-cabundle-cert\") pod \"auth-invalid-token-test-kserve-849f576695-69lfl\" (UID: \"5caf49c5-c902-4acd-949b-753c244a55c9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"
Sep 17 18:20:07.006080 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.005988    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5caf49c5-c902-4acd-949b-753c244a55c9-tmp-dir\") pod \"auth-invalid-token-test-kserve-849f576695-69lfl\" (UID: \"5caf49c5-c902-4acd-949b-753c244a55c9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"
Sep 17 18:20:07.006080 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.006046    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5caf49c5-c902-4acd-949b-753c244a55c9-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-849f576695-69lfl\" (UID: \"5caf49c5-c902-4acd-949b-753c244a55c9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"
Sep 17 18:20:07.006080 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.006070    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b7f6\" (UniqueName: \"kubernetes.io/projected/5caf49c5-c902-4acd-949b-753c244a55c9-kube-api-access-8b7f6\") pod \"auth-invalid-token-test-kserve-849f576695-69lfl\" (UID: \"5caf49c5-c902-4acd-949b-753c244a55c9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"
Sep 17 18:20:07.006253 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.006113    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5caf49c5-c902-4acd-949b-753c244a55c9-home\") pod \"auth-invalid-token-test-kserve-849f576695-69lfl\" (UID: \"5caf49c5-c902-4acd-949b-753c244a55c9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"
Sep 17 18:20:07.095116 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.095082    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg"]
Sep 17 18:20:07.098557 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.098533    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg"
Sep 17 18:20:07.101388 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.101367    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-mbjqg\""
Sep 17 18:20:07.106737 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.106709    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5caf49c5-c902-4acd-949b-753c244a55c9-tls-certs\") pod \"auth-invalid-token-test-kserve-849f576695-69lfl\" (UID: \"5caf49c5-c902-4acd-949b-753c244a55c9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"
Sep 17 18:20:07.106844 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.106805    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5caf49c5-c902-4acd-949b-753c244a55c9-cabundle-cert\") pod \"auth-invalid-token-test-kserve-849f576695-69lfl\" (UID: \"5caf49c5-c902-4acd-949b-753c244a55c9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"
Sep 17 18:20:07.106936 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.106863    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5caf49c5-c902-4acd-949b-753c244a55c9-tmp-dir\") pod \"auth-invalid-token-test-kserve-849f576695-69lfl\" (UID: \"5caf49c5-c902-4acd-949b-753c244a55c9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"
Sep 17 18:20:07.107004 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.106961    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5caf49c5-c902-4acd-949b-753c244a55c9-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-849f576695-69lfl\" (UID: \"5caf49c5-c902-4acd-949b-753c244a55c9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"
Sep 17 18:20:07.107004 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.106992    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b7f6\" (UniqueName: \"kubernetes.io/projected/5caf49c5-c902-4acd-949b-753c244a55c9-kube-api-access-8b7f6\") pod \"auth-invalid-token-test-kserve-849f576695-69lfl\" (UID: \"5caf49c5-c902-4acd-949b-753c244a55c9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"
Sep 17 18:20:07.107111 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.107066    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5caf49c5-c902-4acd-949b-753c244a55c9-home\") pod \"auth-invalid-token-test-kserve-849f576695-69lfl\" (UID: \"5caf49c5-c902-4acd-949b-753c244a55c9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"
Sep 17 18:20:07.107111 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.107098    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg"]
Sep 17 18:20:07.107223 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.107107    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5caf49c5-c902-4acd-949b-753c244a55c9-dshm\") pod \"auth-invalid-token-test-kserve-849f576695-69lfl\" (UID: \"5caf49c5-c902-4acd-949b-753c244a55c9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"
Sep 17 18:20:07.107223 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.107192    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5caf49c5-c902-4acd-949b-753c244a55c9-model-cache\") pod \"auth-invalid-token-test-kserve-849f576695-69lfl\" (UID: \"5caf49c5-c902-4acd-949b-753c244a55c9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"
Sep 17 18:20:07.107515 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.107491    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5caf49c5-c902-4acd-949b-753c244a55c9-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-849f576695-69lfl\" (UID: \"5caf49c5-c902-4acd-949b-753c244a55c9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"
Sep 17 18:20:07.107604 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.107585    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5caf49c5-c902-4acd-949b-753c244a55c9-model-cache\") pod \"auth-invalid-token-test-kserve-849f576695-69lfl\" (UID: \"5caf49c5-c902-4acd-949b-753c244a55c9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"
Sep 17 18:20:07.107685 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.107660    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5caf49c5-c902-4acd-949b-753c244a55c9-cabundle-cert\") pod \"auth-invalid-token-test-kserve-849f576695-69lfl\" (UID: \"5caf49c5-c902-4acd-949b-753c244a55c9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"
Sep 17 18:20:07.108041 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.108019    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5caf49c5-c902-4acd-949b-753c244a55c9-home\") pod \"auth-invalid-token-test-kserve-849f576695-69lfl\" (UID: \"5caf49c5-c902-4acd-949b-753c244a55c9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"
Sep 17 18:20:07.108139 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.108066    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5caf49c5-c902-4acd-949b-753c244a55c9-tmp-dir\") pod \"auth-invalid-token-test-kserve-849f576695-69lfl\" (UID: \"5caf49c5-c902-4acd-949b-753c244a55c9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"
Sep 17 18:20:07.109627 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.109606    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5caf49c5-c902-4acd-949b-753c244a55c9-dshm\") pod \"auth-invalid-token-test-kserve-849f576695-69lfl\" (UID: \"5caf49c5-c902-4acd-949b-753c244a55c9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"
Sep 17 18:20:07.109710 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.109669    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5caf49c5-c902-4acd-949b-753c244a55c9-tls-certs\") pod \"auth-invalid-token-test-kserve-849f576695-69lfl\" (UID: \"5caf49c5-c902-4acd-949b-753c244a55c9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"
Sep 17 18:20:07.120403 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.120372    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b7f6\" (UniqueName: \"kubernetes.io/projected/5caf49c5-c902-4acd-949b-753c244a55c9-kube-api-access-8b7f6\") pod \"auth-invalid-token-test-kserve-849f576695-69lfl\" (UID: \"5caf49c5-c902-4acd-949b-753c244a55c9\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"
Sep 17 18:20:07.198238 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.198143    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"
Sep 17 18:20:07.208592 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.208564    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/552d1695-23ad-4e98-84b5-e743ee6ac9ee-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg\" (UID: \"552d1695-23ad-4e98-84b5-e743ee6ac9ee\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg"
Sep 17 18:20:07.208721 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.208611    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9xc4\" (UniqueName: \"kubernetes.io/projected/552d1695-23ad-4e98-84b5-e743ee6ac9ee-kube-api-access-b9xc4\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg\" (UID: \"552d1695-23ad-4e98-84b5-e743ee6ac9ee\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg"
Sep 17 18:20:07.309325 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.309287    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/552d1695-23ad-4e98-84b5-e743ee6ac9ee-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg\" (UID: \"552d1695-23ad-4e98-84b5-e743ee6ac9ee\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg"
Sep 17 18:20:07.309503 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.309335    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9xc4\" (UniqueName: \"kubernetes.io/projected/552d1695-23ad-4e98-84b5-e743ee6ac9ee-kube-api-access-b9xc4\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg\" (UID: \"552d1695-23ad-4e98-84b5-e743ee6ac9ee\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg"
Sep 17 18:20:07.311746 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.311726    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/552d1695-23ad-4e98-84b5-e743ee6ac9ee-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg\" (UID: \"552d1695-23ad-4e98-84b5-e743ee6ac9ee\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg"
Sep 17 18:20:07.316934 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.316913    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9xc4\" (UniqueName: \"kubernetes.io/projected/552d1695-23ad-4e98-84b5-e743ee6ac9ee-kube-api-access-b9xc4\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg\" (UID: \"552d1695-23ad-4e98-84b5-e743ee6ac9ee\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg"
Sep 17 18:20:07.410963 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.410929    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg"
Sep 17 18:20:07.482882 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.482849    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"]
Sep 17 18:20:07.483819 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:20:07.483787    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5caf49c5_c902_4acd_949b_753c244a55c9.slice/crio-f504c95918153079943e2b75d1b89e36ee56a466bb4f78feaae41b004ac8b0c2 WatchSource:0}: Error finding container f504c95918153079943e2b75d1b89e36ee56a466bb4f78feaae41b004ac8b0c2: Status 404 returned error can't find the container with id f504c95918153079943e2b75d1b89e36ee56a466bb4f78feaae41b004ac8b0c2
Sep 17 18:20:07.666609 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.666567    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl" event={"ID":"5caf49c5-c902-4acd-949b-753c244a55c9","Type":"ContainerStarted","Data":"28d46cec8fc2a518dc88305633527d7a567841abf6d650fbff72fff17fba88d1"}
Sep 17 18:20:07.666918 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.666632    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl" event={"ID":"5caf49c5-c902-4acd-949b-753c244a55c9","Type":"ContainerStarted","Data":"f504c95918153079943e2b75d1b89e36ee56a466bb4f78feaae41b004ac8b0c2"}
Sep 17 18:20:07.685379 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:07.685349    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg"]
Sep 17 18:20:07.688212 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:20:07.688162    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod552d1695_23ad_4e98_84b5_e743ee6ac9ee.slice/crio-d7831d7ca569eae440ed399ee3446f34626e9dc319959fae42b7ec018eaf6614 WatchSource:0}: Error finding container d7831d7ca569eae440ed399ee3446f34626e9dc319959fae42b7ec018eaf6614: Status 404 returned error can't find the container with id d7831d7ca569eae440ed399ee3446f34626e9dc319959fae42b7ec018eaf6614
Sep 17 18:20:08.673671 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:08.673631    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg" event={"ID":"552d1695-23ad-4e98-84b5-e743ee6ac9ee","Type":"ContainerStarted","Data":"315f487c9c2f64666d1ea7d54afa843cf60a59a1d8154674eb308cbab3247f57"}
Sep 17 18:20:08.674134 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:08.673695    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg" event={"ID":"552d1695-23ad-4e98-84b5-e743ee6ac9ee","Type":"ContainerStarted","Data":"d7831d7ca569eae440ed399ee3446f34626e9dc319959fae42b7ec018eaf6614"}
Sep 17 18:20:08.674134 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:08.673932    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg"
Sep 17 18:20:08.692402 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:08.692352    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg" podStartSLOduration=1.69233295 podStartE2EDuration="1.69233295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:20:08.691034932 +0000 UTC m=+1396.781656587" watchObservedRunningTime="2026-09-17 18:20:08.69233295 +0000 UTC m=+1396.782954599"
Sep 17 18:20:10.685463 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:10.685427    2581 generic.go:356] "Generic (PLEG): container finished" podID="5caf49c5-c902-4acd-949b-753c244a55c9" containerID="28d46cec8fc2a518dc88305633527d7a567841abf6d650fbff72fff17fba88d1" exitCode=0
Sep 17 18:20:10.685869 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:10.685503    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl" event={"ID":"5caf49c5-c902-4acd-949b-753c244a55c9","Type":"ContainerDied","Data":"28d46cec8fc2a518dc88305633527d7a567841abf6d650fbff72fff17fba88d1"}
Sep 17 18:20:10.686873 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:10.686858    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:20:38.872218 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:38.872109    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl" event={"ID":"5caf49c5-c902-4acd-949b-753c244a55c9","Type":"ContainerStarted","Data":"b08b954c65eb92a1609a3f489aef672283ad4b4621288fafafc0a34cccdf9c76"}
Sep 17 18:20:38.896917 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:38.896843    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl" podStartSLOduration=2.729710266 podStartE2EDuration="32.896826786s" totalImagesPullingTime="27.80989519s" totalInitContainerRuntime="2.35722133s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:20:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:20:38.894560483 +0000 UTC m=+1426.985182134" watchObservedRunningTime="2026-09-17 18:20:38.896826786 +0000 UTC m=+1426.987448435"
Sep 17 18:20:39.198885 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:39.198806    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"
Sep 17 18:20:39.680404 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:39.680377    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg"
Sep 17 18:20:44.957585 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:44.957552    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"]
Sep 17 18:20:44.960363 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:44.960330    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m"]
Sep 17 18:20:44.971951 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:20:44.971892    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:20:44.971951 ip-10-0-130-28 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:20:44.971951 ip-10-0-130-28 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:20:44.971951 ip-10-0-130-28 kubenswrapper[2581]:  > containerID="3fffabdfb494f25fd9452f3cf05f2174466ea9ea282ce4a1491b48d40540c32d" cmd=["/bin/sleep","15"]
Sep 17 18:20:44.972298 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:44.971972    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl" message="" err=<
Sep 17 18:20:44.972298 ip-10-0-130-28 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:20:44.972298 ip-10-0-130-28 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:20:44.972298 ip-10-0-130-28 kubenswrapper[2581]:  >
Sep 17 18:20:44.972298 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:20:44.971989    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:20:44.972298 ip-10-0-130-28 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:20:44.972298 ip-10-0-130-28 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:20:44.972298 ip-10-0-130-28 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl" podUID="8fc343e6-bb26-4c06-a6c4-daa249f05175" containerName="main" containerID="cri-o://3fffabdfb494f25fd9452f3cf05f2174466ea9ea282ce4a1491b48d40540c32d"
Sep 17 18:20:44.972298 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:44.972049    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl" podUID="8fc343e6-bb26-4c06-a6c4-daa249f05175" containerName="main" containerID="cri-o://3fffabdfb494f25fd9452f3cf05f2174466ea9ea282ce4a1491b48d40540c32d" gracePeriod=60
Sep 17 18:20:45.500065 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:45.500023    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl" podUID="8fc343e6-bb26-4c06-a6c4-daa249f05175" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Sep 17 18:20:45.786477 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:45.786442    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"
Sep 17 18:20:45.904020 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:45.903927    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fc343e6-bb26-4c06-a6c4-daa249f05175-kserve-provision-location\") pod \"8fc343e6-bb26-4c06-a6c4-daa249f05175\" (UID: \"8fc343e6-bb26-4c06-a6c4-daa249f05175\") "
Sep 17 18:20:45.904020 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:45.903980    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc343e6-bb26-4c06-a6c4-daa249f05175-dshm\") pod \"8fc343e6-bb26-4c06-a6c4-daa249f05175\" (UID: \"8fc343e6-bb26-4c06-a6c4-daa249f05175\") "
Sep 17 18:20:45.904020 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:45.904016    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fc343e6-bb26-4c06-a6c4-daa249f05175-cabundle-cert\") pod \"8fc343e6-bb26-4c06-a6c4-daa249f05175\" (UID: \"8fc343e6-bb26-4c06-a6c4-daa249f05175\") "
Sep 17 18:20:45.904313 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:45.904063    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc343e6-bb26-4c06-a6c4-daa249f05175-home\") pod \"8fc343e6-bb26-4c06-a6c4-daa249f05175\" (UID: \"8fc343e6-bb26-4c06-a6c4-daa249f05175\") "
Sep 17 18:20:45.904313 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:45.904095    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc343e6-bb26-4c06-a6c4-daa249f05175-tmp-dir\") pod \"8fc343e6-bb26-4c06-a6c4-daa249f05175\" (UID: \"8fc343e6-bb26-4c06-a6c4-daa249f05175\") "
Sep 17 18:20:45.904313 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:45.904123    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc343e6-bb26-4c06-a6c4-daa249f05175-model-cache\") pod \"8fc343e6-bb26-4c06-a6c4-daa249f05175\" (UID: \"8fc343e6-bb26-4c06-a6c4-daa249f05175\") "
Sep 17 18:20:45.904313 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:45.904159    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc343e6-bb26-4c06-a6c4-daa249f05175-tls-certs\") pod \"8fc343e6-bb26-4c06-a6c4-daa249f05175\" (UID: \"8fc343e6-bb26-4c06-a6c4-daa249f05175\") "
Sep 17 18:20:45.904313 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:45.904190    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrr88\" (UniqueName: \"kubernetes.io/projected/8fc343e6-bb26-4c06-a6c4-daa249f05175-kube-api-access-qrr88\") pod \"8fc343e6-bb26-4c06-a6c4-daa249f05175\" (UID: \"8fc343e6-bb26-4c06-a6c4-daa249f05175\") "
Sep 17 18:20:45.904553 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:45.904298    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc343e6-bb26-4c06-a6c4-daa249f05175-home" (OuterVolumeSpecName: "home") pod "8fc343e6-bb26-4c06-a6c4-daa249f05175" (UID: "8fc343e6-bb26-4c06-a6c4-daa249f05175"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:20:45.904553 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:45.904365    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fc343e6-bb26-4c06-a6c4-daa249f05175-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8fc343e6-bb26-4c06-a6c4-daa249f05175" (UID: "8fc343e6-bb26-4c06-a6c4-daa249f05175"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:20:45.904650 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:45.904556    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fc343e6-bb26-4c06-a6c4-daa249f05175-cabundle-cert\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:20:45.904650 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:45.904575    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc343e6-bb26-4c06-a6c4-daa249f05175-home\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:20:45.904959 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:45.904917    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc343e6-bb26-4c06-a6c4-daa249f05175-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8fc343e6-bb26-4c06-a6c4-daa249f05175" (UID: "8fc343e6-bb26-4c06-a6c4-daa249f05175"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:20:45.905137 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:45.905113    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc343e6-bb26-4c06-a6c4-daa249f05175-model-cache" (OuterVolumeSpecName: "model-cache") pod "8fc343e6-bb26-4c06-a6c4-daa249f05175" (UID: "8fc343e6-bb26-4c06-a6c4-daa249f05175"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:20:45.906979 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:45.906949    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc343e6-bb26-4c06-a6c4-daa249f05175-dshm" (OuterVolumeSpecName: "dshm") pod "8fc343e6-bb26-4c06-a6c4-daa249f05175" (UID: "8fc343e6-bb26-4c06-a6c4-daa249f05175"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:20:45.907106 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:45.907029    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fc343e6-bb26-4c06-a6c4-daa249f05175-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8fc343e6-bb26-4c06-a6c4-daa249f05175" (UID: "8fc343e6-bb26-4c06-a6c4-daa249f05175"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:20:45.907106 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:45.907040    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc343e6-bb26-4c06-a6c4-daa249f05175-kube-api-access-qrr88" (OuterVolumeSpecName: "kube-api-access-qrr88") pod "8fc343e6-bb26-4c06-a6c4-daa249f05175" (UID: "8fc343e6-bb26-4c06-a6c4-daa249f05175"). InnerVolumeSpecName "kube-api-access-qrr88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:20:45.919865 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:45.919833    2581 generic.go:356] "Generic (PLEG): container finished" podID="8fc343e6-bb26-4c06-a6c4-daa249f05175" containerID="3fffabdfb494f25fd9452f3cf05f2174466ea9ea282ce4a1491b48d40540c32d" exitCode=0
Sep 17 18:20:45.920024 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:45.919880    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"
Sep 17 18:20:45.920024 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:45.919937    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl" event={"ID":"8fc343e6-bb26-4c06-a6c4-daa249f05175","Type":"ContainerDied","Data":"3fffabdfb494f25fd9452f3cf05f2174466ea9ea282ce4a1491b48d40540c32d"}
Sep 17 18:20:45.920024 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:45.919994    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl" event={"ID":"8fc343e6-bb26-4c06-a6c4-daa249f05175","Type":"ContainerDied","Data":"2d83606114109791357ecd2c28eb279ee6b0dc1d6578dc1b6eb16ddca823178b"}
Sep 17 18:20:45.920024 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:45.920024    2581 scope.go:117] "RemoveContainer" containerID="3fffabdfb494f25fd9452f3cf05f2174466ea9ea282ce4a1491b48d40540c32d"
Sep 17 18:20:45.933391 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:45.933350    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc343e6-bb26-4c06-a6c4-daa249f05175-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8fc343e6-bb26-4c06-a6c4-daa249f05175" (UID: "8fc343e6-bb26-4c06-a6c4-daa249f05175"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:20:45.934721 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:45.934701    2581 scope.go:117] "RemoveContainer" containerID="650f1f58809d46c9a7ad0aecc4bb9bcfc6e02630ea70f05cac98245f1311bd62"
Sep 17 18:20:45.943701 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:45.943677    2581 scope.go:117] "RemoveContainer" containerID="3fffabdfb494f25fd9452f3cf05f2174466ea9ea282ce4a1491b48d40540c32d"
Sep 17 18:20:45.944057 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:20:45.944025    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fffabdfb494f25fd9452f3cf05f2174466ea9ea282ce4a1491b48d40540c32d\": container with ID starting with 3fffabdfb494f25fd9452f3cf05f2174466ea9ea282ce4a1491b48d40540c32d not found: ID does not exist" containerID="3fffabdfb494f25fd9452f3cf05f2174466ea9ea282ce4a1491b48d40540c32d"
Sep 17 18:20:45.944165 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:45.944064    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fffabdfb494f25fd9452f3cf05f2174466ea9ea282ce4a1491b48d40540c32d"} err="failed to get container status \"3fffabdfb494f25fd9452f3cf05f2174466ea9ea282ce4a1491b48d40540c32d\": rpc error: code = NotFound desc = could not find container \"3fffabdfb494f25fd9452f3cf05f2174466ea9ea282ce4a1491b48d40540c32d\": container with ID starting with 3fffabdfb494f25fd9452f3cf05f2174466ea9ea282ce4a1491b48d40540c32d not found: ID does not exist"
Sep 17 18:20:45.944165 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:45.944082    2581 scope.go:117] "RemoveContainer" containerID="650f1f58809d46c9a7ad0aecc4bb9bcfc6e02630ea70f05cac98245f1311bd62"
Sep 17 18:20:45.944388 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:20:45.944367    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"650f1f58809d46c9a7ad0aecc4bb9bcfc6e02630ea70f05cac98245f1311bd62\": container with ID starting with 650f1f58809d46c9a7ad0aecc4bb9bcfc6e02630ea70f05cac98245f1311bd62 not found: ID does not exist" containerID="650f1f58809d46c9a7ad0aecc4bb9bcfc6e02630ea70f05cac98245f1311bd62"
Sep 17 18:20:45.944453 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:45.944396    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"650f1f58809d46c9a7ad0aecc4bb9bcfc6e02630ea70f05cac98245f1311bd62"} err="failed to get container status \"650f1f58809d46c9a7ad0aecc4bb9bcfc6e02630ea70f05cac98245f1311bd62\": rpc error: code = NotFound desc = could not find container \"650f1f58809d46c9a7ad0aecc4bb9bcfc6e02630ea70f05cac98245f1311bd62\": container with ID starting with 650f1f58809d46c9a7ad0aecc4bb9bcfc6e02630ea70f05cac98245f1311bd62 not found: ID does not exist"
Sep 17 18:20:46.005500 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:46.005465    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc343e6-bb26-4c06-a6c4-daa249f05175-model-cache\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:20:46.005500 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:46.005495    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc343e6-bb26-4c06-a6c4-daa249f05175-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:20:46.005500 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:46.005505    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrr88\" (UniqueName: \"kubernetes.io/projected/8fc343e6-bb26-4c06-a6c4-daa249f05175-kube-api-access-qrr88\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:20:46.005961 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:46.005515    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fc343e6-bb26-4c06-a6c4-daa249f05175-kserve-provision-location\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:20:46.005961 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:46.005525    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc343e6-bb26-4c06-a6c4-daa249f05175-dshm\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:20:46.005961 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:46.005534    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc343e6-bb26-4c06-a6c4-daa249f05175-tmp-dir\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:20:46.242291 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:46.242252    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"]
Sep 17 18:20:46.245756 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:46.245725    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-7bc6df44dc-hrpsl"]
Sep 17 18:20:46.358555 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:46.358518    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fc343e6-bb26-4c06-a6c4-daa249f05175" path="/var/lib/kubelet/pods/8fc343e6-bb26-4c06-a6c4-daa249f05175/volumes"
Sep 17 18:20:47.198323 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:47.198275    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"
Sep 17 18:20:47.199794 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:47.199752    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl" podUID="5caf49c5-c902-4acd-949b-753c244a55c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 18:20:53.447762 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:53.447720    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m" podUID="0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:20:57.198835 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:57.198780    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl" podUID="5caf49c5-c902-4acd-949b-753c244a55c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 18:20:59.982754 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:20:59.982677    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m" podUID="0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb" containerName="main" containerID="cri-o://70d4d50bce2f75e2afdf5191004ba0e37b799196f5f2790194cb069520ab16bb" gracePeriod=45
Sep 17 18:21:03.447942 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:03.447885    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m" podUID="0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:21:07.198705 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:07.198660    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl" podUID="5caf49c5-c902-4acd-949b-753c244a55c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 18:21:13.448119 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:13.448077    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m" podUID="0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:21:13.448619 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:13.448244    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m"
Sep 17 18:21:17.199658 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:17.199609    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl" podUID="5caf49c5-c902-4acd-949b-753c244a55c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 18:21:23.447698 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:23.447651    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m" podUID="0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:21:27.199419 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:27.199366    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl" podUID="5caf49c5-c902-4acd-949b-753c244a55c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 18:21:30.148646 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:30.148607    2581 generic.go:356] "Generic (PLEG): container finished" podID="0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb" containerID="70d4d50bce2f75e2afdf5191004ba0e37b799196f5f2790194cb069520ab16bb" exitCode=0
Sep 17 18:21:30.149119 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:30.148663    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m" event={"ID":"0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb","Type":"ContainerDied","Data":"70d4d50bce2f75e2afdf5191004ba0e37b799196f5f2790194cb069520ab16bb"}
Sep 17 18:21:30.233422 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:30.233393    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m"
Sep 17 18:21:30.312425 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:30.312340    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7nth\" (UniqueName: \"kubernetes.io/projected/0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb-kube-api-access-w7nth\") pod \"0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb\" (UID: \"0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb\") "
Sep 17 18:21:30.312607 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:30.312444    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb-tls-certs\") pod \"0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb\" (UID: \"0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb\") "
Sep 17 18:21:30.314399 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:30.314367    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb-kube-api-access-w7nth" (OuterVolumeSpecName: "kube-api-access-w7nth") pod "0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb" (UID: "0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb"). InnerVolumeSpecName "kube-api-access-w7nth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:21:30.314399 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:30.314387    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb" (UID: "0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:21:30.414158 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:30.414123    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:21:30.414158 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:30.414153    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7nth\" (UniqueName: \"kubernetes.io/projected/0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb-kube-api-access-w7nth\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:21:31.155354 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:31.155326    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m"
Sep 17 18:21:31.155782 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:31.155324    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m" event={"ID":"0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb","Type":"ContainerDied","Data":"ab20cf924b6654a8a26050af90daac9107b19d922f86e07bce18a88082c1f658"}
Sep 17 18:21:31.155782 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:31.155459    2581 scope.go:117] "RemoveContainer" containerID="70d4d50bce2f75e2afdf5191004ba0e37b799196f5f2790194cb069520ab16bb"
Sep 17 18:21:31.173156 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:31.173131    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m"]
Sep 17 18:21:31.179482 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:31.179441    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d5p5m"]
Sep 17 18:21:32.359115 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:32.359076    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb" path="/var/lib/kubelet/pods/0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb/volumes"
Sep 17 18:21:37.198779 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:37.198730    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl" podUID="5caf49c5-c902-4acd-949b-753c244a55c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 18:21:43.994143 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:43.993962    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"]
Sep 17 18:21:43.994610 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:43.994542    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fc343e6-bb26-4c06-a6c4-daa249f05175" containerName="storage-initializer"
Sep 17 18:21:43.994610 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:43.994562    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fc343e6-bb26-4c06-a6c4-daa249f05175" containerName="storage-initializer"
Sep 17 18:21:43.994610 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:43.994597    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fc343e6-bb26-4c06-a6c4-daa249f05175" containerName="main"
Sep 17 18:21:43.994610 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:43.994603    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fc343e6-bb26-4c06-a6c4-daa249f05175" containerName="main"
Sep 17 18:21:43.994755 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:43.994615    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb" containerName="main"
Sep 17 18:21:43.994755 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:43.994625    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb" containerName="main"
Sep 17 18:21:43.994755 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:43.994690    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ea8de85-17e6-4bc2-8ffb-a941d49a5aeb" containerName="main"
Sep 17 18:21:43.994755 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:43.994706    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fc343e6-bb26-4c06-a6c4-daa249f05175" containerName="main"
Sep 17 18:21:43.997717 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:43.997699    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"
Sep 17 18:21:44.000449 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.000424    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 17 18:21:44.001501 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.001482    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 17 18:21:44.001756 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.001737    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 17 18:21:44.001826 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.001768    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:21:44.001826 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.001772    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-6tcbp\""
Sep 17 18:21:44.001950 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.001818    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 17 18:21:44.006773 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.006752    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"]
Sep 17 18:21:44.144982 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.144885    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-model-cache\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-4p66d\" (UID: \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"
Sep 17 18:21:44.145170 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.145036    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-4p66d\" (UID: \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"
Sep 17 18:21:44.145170 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.145074    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-home\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-4p66d\" (UID: \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"
Sep 17 18:21:44.145170 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.145096    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-dshm\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-4p66d\" (UID: \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"
Sep 17 18:21:44.145346 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.145182    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctm6d\" (UniqueName: \"kubernetes.io/projected/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-kube-api-access-ctm6d\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-4p66d\" (UID: \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"
Sep 17 18:21:44.145346 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.145267    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-4p66d\" (UID: \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"
Sep 17 18:21:44.145346 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.145296    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-4p66d\" (UID: \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"
Sep 17 18:21:44.145346 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.145320    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-4p66d\" (UID: \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"
Sep 17 18:21:44.234051 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.234019    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7"]
Sep 17 18:21:44.236701 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.236665    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7"
Sep 17 18:21:44.239818 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.239795    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-7m44r\""
Sep 17 18:21:44.245976 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.245812    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-4p66d\" (UID: \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"
Sep 17 18:21:44.245976 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.245866    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-home\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-4p66d\" (UID: \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"
Sep 17 18:21:44.245976 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.245886    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-dshm\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-4p66d\" (UID: \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"
Sep 17 18:21:44.245976 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.245974    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctm6d\" (UniqueName: \"kubernetes.io/projected/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-kube-api-access-ctm6d\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-4p66d\" (UID: \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"
Sep 17 18:21:44.246274 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.246035    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-4p66d\" (UID: \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"
Sep 17 18:21:44.246274 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.246065    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-4p66d\" (UID: \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"
Sep 17 18:21:44.246274 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.246086    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-4p66d\" (UID: \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"
Sep 17 18:21:44.246274 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.246124    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-model-cache\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-4p66d\" (UID: \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"
Sep 17 18:21:44.246513 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.246352    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-home\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-4p66d\" (UID: \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"
Sep 17 18:21:44.246513 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.246378    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-4p66d\" (UID: \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"
Sep 17 18:21:44.246513 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.246404    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-model-cache\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-4p66d\" (UID: \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"
Sep 17 18:21:44.246678 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.246595    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-4p66d\" (UID: \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"
Sep 17 18:21:44.247046 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.247021    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-4p66d\" (UID: \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"
Sep 17 18:21:44.249174 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.249152    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7"]
Sep 17 18:21:44.249330 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.249310    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-4p66d\" (UID: \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"
Sep 17 18:21:44.249638 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.249615    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-dshm\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-4p66d\" (UID: \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"
Sep 17 18:21:44.255165 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.255008    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctm6d\" (UniqueName: \"kubernetes.io/projected/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-kube-api-access-ctm6d\") pod \"tokenizer-migration-path-test-kserve-7b8d65d846-4p66d\" (UID: \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"
Sep 17 18:21:44.311019 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.310983    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"
Sep 17 18:21:44.347211 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.347175    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r7gd\" (UniqueName: \"kubernetes.io/projected/2fc77015-7cba-4972-bf20-56aa16bbec21-kube-api-access-2r7gd\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7\" (UID: \"2fc77015-7cba-4972-bf20-56aa16bbec21\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7"
Sep 17 18:21:44.347360 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.347246    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fc77015-7cba-4972-bf20-56aa16bbec21-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7\" (UID: \"2fc77015-7cba-4972-bf20-56aa16bbec21\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7"
Sep 17 18:21:44.448680 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.448634    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r7gd\" (UniqueName: \"kubernetes.io/projected/2fc77015-7cba-4972-bf20-56aa16bbec21-kube-api-access-2r7gd\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7\" (UID: \"2fc77015-7cba-4972-bf20-56aa16bbec21\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7"
Sep 17 18:21:44.448890 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.448715    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fc77015-7cba-4972-bf20-56aa16bbec21-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7\" (UID: \"2fc77015-7cba-4972-bf20-56aa16bbec21\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7"
Sep 17 18:21:44.451303 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.451270    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fc77015-7cba-4972-bf20-56aa16bbec21-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7\" (UID: \"2fc77015-7cba-4972-bf20-56aa16bbec21\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7"
Sep 17 18:21:44.457513 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.457481    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r7gd\" (UniqueName: \"kubernetes.io/projected/2fc77015-7cba-4972-bf20-56aa16bbec21-kube-api-access-2r7gd\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7\" (UID: \"2fc77015-7cba-4972-bf20-56aa16bbec21\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7"
Sep 17 18:21:44.549434 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.549335    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7"
Sep 17 18:21:44.597344 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.597257    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"]
Sep 17 18:21:44.827419 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:44.827381    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7"]
Sep 17 18:21:44.830356 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:21:44.830323    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc77015_7cba_4972_bf20_56aa16bbec21.slice/crio-b117964862cfc6f495d6cab174c84af0e319c57e697b6515f51ac6105447d2b7 WatchSource:0}: Error finding container b117964862cfc6f495d6cab174c84af0e319c57e697b6515f51ac6105447d2b7: Status 404 returned error can't find the container with id b117964862cfc6f495d6cab174c84af0e319c57e697b6515f51ac6105447d2b7
Sep 17 18:21:45.229183 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:45.229135    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7" event={"ID":"2fc77015-7cba-4972-bf20-56aa16bbec21","Type":"ContainerStarted","Data":"449886dc912917f9ef9a08aafbafb32fb083d5806b6bba53a6d6af6a48675bd1"}
Sep 17 18:21:45.229591 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:45.229209    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7" event={"ID":"2fc77015-7cba-4972-bf20-56aa16bbec21","Type":"ContainerStarted","Data":"b117964862cfc6f495d6cab174c84af0e319c57e697b6515f51ac6105447d2b7"}
Sep 17 18:21:45.230068 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:45.230046    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7"
Sep 17 18:21:45.232232 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:45.232189    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d" event={"ID":"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a","Type":"ContainerStarted","Data":"5790d7697f529f9744ca521b84b953f1c00f1a0284c17362b6fda5f4098126e3"}
Sep 17 18:21:45.232370 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:45.232252    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d" event={"ID":"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a","Type":"ContainerStarted","Data":"d6acc9f9e7eca4f510637b42d989d1fb52e2d03dd7fea92af288d754abd38a64"}
Sep 17 18:21:45.251861 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:45.251791    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7" podStartSLOduration=1.251770856 podStartE2EDuration="1.251770856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:21:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:21:45.249116753 +0000 UTC m=+1493.339738403" watchObservedRunningTime="2026-09-17 18:21:45.251770856 +0000 UTC m=+1493.342392506"
Sep 17 18:21:47.198923 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:47.198871    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl" podUID="5caf49c5-c902-4acd-949b-753c244a55c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 18:21:48.251854 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:48.251823    2581 generic.go:356] "Generic (PLEG): container finished" podID="8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a" containerID="5790d7697f529f9744ca521b84b953f1c00f1a0284c17362b6fda5f4098126e3" exitCode=0
Sep 17 18:21:48.252286 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:48.251918    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d" event={"ID":"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a","Type":"ContainerDied","Data":"5790d7697f529f9744ca521b84b953f1c00f1a0284c17362b6fda5f4098126e3"}
Sep 17 18:21:49.259120 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:49.259072    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d" event={"ID":"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a","Type":"ContainerStarted","Data":"14a40aca386249c36c0b8a53dc7798fccbf57e9cca7adede2e6bdb45d32ce8d4"}
Sep 17 18:21:49.279446 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:49.279393    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d" podStartSLOduration=3.560615241 podStartE2EDuration="6.279375356s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.718760115s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:21:49.278338651 +0000 UTC m=+1497.368960302" watchObservedRunningTime="2026-09-17 18:21:49.279375356 +0000 UTC m=+1497.369997005"
Sep 17 18:21:49.311632 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:49.311591    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"
Sep 17 18:21:52.418689 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:52.418659    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/2.log"
Sep 17 18:21:52.423517 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:52.423488    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/2.log"
Sep 17 18:21:54.311669 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:54.311627    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"
Sep 17 18:21:54.324363 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:54.324335    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"
Sep 17 18:21:55.305881 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:55.305852    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"
Sep 17 18:21:57.199133 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:21:57.199091    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl" podUID="5caf49c5-c902-4acd-949b-753c244a55c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 18:22:07.199021 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:07.198978    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl" podUID="5caf49c5-c902-4acd-949b-753c244a55c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 18:22:17.198997 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:17.198884    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl" podUID="5caf49c5-c902-4acd-949b-753c244a55c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 18:22:17.245670 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:17.245641    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7"
Sep 17 18:22:21.404693 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:21.404650    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7"]
Sep 17 18:22:21.414648 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:21.413531    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"]
Sep 17 18:22:21.428087 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:22:21.427802    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:22:21.428087 ip-10-0-130-28 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:22:21.428087 ip-10-0-130-28 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:22:21.428087 ip-10-0-130-28 kubenswrapper[2581]:  > containerID="14a40aca386249c36c0b8a53dc7798fccbf57e9cca7adede2e6bdb45d32ce8d4" cmd=["/bin/sleep","15"]
Sep 17 18:22:21.428087 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:21.427864    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d" message="" err=<
Sep 17 18:22:21.428087 ip-10-0-130-28 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:22:21.428087 ip-10-0-130-28 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:22:21.428087 ip-10-0-130-28 kubenswrapper[2581]:  >
Sep 17 18:22:21.428087 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:22:21.427879    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:22:21.428087 ip-10-0-130-28 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:22:21.428087 ip-10-0-130-28 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:22:21.428087 ip-10-0-130-28 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d" podUID="8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a" containerName="main" containerID="cri-o://14a40aca386249c36c0b8a53dc7798fccbf57e9cca7adede2e6bdb45d32ce8d4"
Sep 17 18:22:21.428087 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:21.427960    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d" podUID="8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a" containerName="main" containerID="cri-o://14a40aca386249c36c0b8a53dc7798fccbf57e9cca7adede2e6bdb45d32ce8d4" gracePeriod=60
Sep 17 18:22:21.778754 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:21.778717    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"
Sep 17 18:22:21.892355 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:21.892320    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-model-cache\") pod \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\" (UID: \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\") "
Sep 17 18:22:21.892555 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:21.892391    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-kserve-provision-location\") pod \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\" (UID: \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\") "
Sep 17 18:22:21.892555 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:21.892450    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctm6d\" (UniqueName: \"kubernetes.io/projected/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-kube-api-access-ctm6d\") pod \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\" (UID: \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\") "
Sep 17 18:22:21.892555 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:21.892495    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-cabundle-cert\") pod \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\" (UID: \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\") "
Sep 17 18:22:21.892555 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:21.892542    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-home\") pod \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\" (UID: \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\") "
Sep 17 18:22:21.892780 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:21.892563    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-dshm\") pod \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\" (UID: \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\") "
Sep 17 18:22:21.892780 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:21.892630    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-model-cache" (OuterVolumeSpecName: "model-cache") pod "8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a" (UID: "8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:21.892897 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:21.892813    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-tls-certs\") pod \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\" (UID: \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\") "
Sep 17 18:22:21.892897 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:21.892865    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-tmp-dir\") pod \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\" (UID: \"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a\") "
Sep 17 18:22:21.892897 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:21.892879    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a" (UID: "8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:22:21.893076 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:21.893011    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-home" (OuterVolumeSpecName: "home") pod "8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a" (UID: "8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:21.893076 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:21.893062    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a" (UID: "8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:21.893325 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:21.893291    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-model-cache\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:22:21.893325 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:21.893313    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-cabundle-cert\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:22:21.893325 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:21.893326    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-home\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:22:21.893554 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:21.893336    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-tmp-dir\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:22:21.895438 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:21.895401    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-dshm" (OuterVolumeSpecName: "dshm") pod "8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a" (UID: "8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:21.895438 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:21.895420    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a" (UID: "8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:22:21.895567 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:21.895513    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-kube-api-access-ctm6d" (OuterVolumeSpecName: "kube-api-access-ctm6d") pod "8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a" (UID: "8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a"). InnerVolumeSpecName "kube-api-access-ctm6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:22:21.924301 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:21.924267    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a" (UID: "8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:21.994015 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:21.993980    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctm6d\" (UniqueName: \"kubernetes.io/projected/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-kube-api-access-ctm6d\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:22:21.994015 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:21.994011    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-dshm\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:22:21.994015 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:21.994022    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:22:21.994253 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:21.994031    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a-kserve-provision-location\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:22:22.425122 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:22.425084    2581 generic.go:356] "Generic (PLEG): container finished" podID="8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a" containerID="14a40aca386249c36c0b8a53dc7798fccbf57e9cca7adede2e6bdb45d32ce8d4" exitCode=0
Sep 17 18:22:22.425503 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:22.425133    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"
Sep 17 18:22:22.425503 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:22.425145    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d" event={"ID":"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a","Type":"ContainerDied","Data":"14a40aca386249c36c0b8a53dc7798fccbf57e9cca7adede2e6bdb45d32ce8d4"}
Sep 17 18:22:22.425503 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:22.425197    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d" event={"ID":"8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a","Type":"ContainerDied","Data":"d6acc9f9e7eca4f510637b42d989d1fb52e2d03dd7fea92af288d754abd38a64"}
Sep 17 18:22:22.425503 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:22.425235    2581 scope.go:117] "RemoveContainer" containerID="14a40aca386249c36c0b8a53dc7798fccbf57e9cca7adede2e6bdb45d32ce8d4"
Sep 17 18:22:22.434164 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:22.434025    2581 scope.go:117] "RemoveContainer" containerID="5790d7697f529f9744ca521b84b953f1c00f1a0284c17362b6fda5f4098126e3"
Sep 17 18:22:22.443573 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:22.443554    2581 scope.go:117] "RemoveContainer" containerID="14a40aca386249c36c0b8a53dc7798fccbf57e9cca7adede2e6bdb45d32ce8d4"
Sep 17 18:22:22.443874 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:22:22.443853    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a40aca386249c36c0b8a53dc7798fccbf57e9cca7adede2e6bdb45d32ce8d4\": container with ID starting with 14a40aca386249c36c0b8a53dc7798fccbf57e9cca7adede2e6bdb45d32ce8d4 not found: ID does not exist" containerID="14a40aca386249c36c0b8a53dc7798fccbf57e9cca7adede2e6bdb45d32ce8d4"
Sep 17 18:22:22.443961 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:22.443883    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a40aca386249c36c0b8a53dc7798fccbf57e9cca7adede2e6bdb45d32ce8d4"} err="failed to get container status \"14a40aca386249c36c0b8a53dc7798fccbf57e9cca7adede2e6bdb45d32ce8d4\": rpc error: code = NotFound desc = could not find container \"14a40aca386249c36c0b8a53dc7798fccbf57e9cca7adede2e6bdb45d32ce8d4\": container with ID starting with 14a40aca386249c36c0b8a53dc7798fccbf57e9cca7adede2e6bdb45d32ce8d4 not found: ID does not exist"
Sep 17 18:22:22.443961 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:22.443921    2581 scope.go:117] "RemoveContainer" containerID="5790d7697f529f9744ca521b84b953f1c00f1a0284c17362b6fda5f4098126e3"
Sep 17 18:22:22.444196 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:22:22.444179    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5790d7697f529f9744ca521b84b953f1c00f1a0284c17362b6fda5f4098126e3\": container with ID starting with 5790d7697f529f9744ca521b84b953f1c00f1a0284c17362b6fda5f4098126e3 not found: ID does not exist" containerID="5790d7697f529f9744ca521b84b953f1c00f1a0284c17362b6fda5f4098126e3"
Sep 17 18:22:22.444258 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:22.444200    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5790d7697f529f9744ca521b84b953f1c00f1a0284c17362b6fda5f4098126e3"} err="failed to get container status \"5790d7697f529f9744ca521b84b953f1c00f1a0284c17362b6fda5f4098126e3\": rpc error: code = NotFound desc = could not find container \"5790d7697f529f9744ca521b84b953f1c00f1a0284c17362b6fda5f4098126e3\": container with ID starting with 5790d7697f529f9744ca521b84b953f1c00f1a0284c17362b6fda5f4098126e3 not found: ID does not exist"
Sep 17 18:22:22.444819 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:22.444801    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"]
Sep 17 18:22:22.450488 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:22.450468    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7b8d65d846-4p66d"]
Sep 17 18:22:24.357721 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:24.357688    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a" path="/var/lib/kubelet/pods/8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a/volumes"
Sep 17 18:22:27.199443 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:27.199402    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl" podUID="5caf49c5-c902-4acd-949b-753c244a55c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 18:22:27.246472 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:27.246433    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7" podUID="2fc77015-7cba-4972-bf20-56aa16bbec21" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:22:36.433431 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:36.433391    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7" podUID="2fc77015-7cba-4972-bf20-56aa16bbec21" containerName="main" containerID="cri-o://449886dc912917f9ef9a08aafbafb32fb083d5806b6bba53a6d6af6a48675bd1" gracePeriod=45
Sep 17 18:22:37.209446 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:37.209411    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"
Sep 17 18:22:37.217069 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:37.217044    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"
Sep 17 18:22:37.245576 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:37.245534    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7" podUID="2fc77015-7cba-4972-bf20-56aa16bbec21" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:22:38.715037 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:38.715001    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg"]
Sep 17 18:22:38.724719 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:38.724684    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"]
Sep 17 18:22:39.680294 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:39.680251    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg" podUID="552d1695-23ad-4e98-84b5-e743ee6ac9ee" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:22:47.246302 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:47.246256    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7" podUID="2fc77015-7cba-4972-bf20-56aa16bbec21" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:22:47.246685 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:47.246371    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7"
Sep 17 18:22:49.680422 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:49.680381    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg" podUID="552d1695-23ad-4e98-84b5-e743ee6ac9ee" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:22:53.735998 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:53.735954    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg" podUID="552d1695-23ad-4e98-84b5-e743ee6ac9ee" containerName="main" containerID="cri-o://315f487c9c2f64666d1ea7d54afa843cf60a59a1d8154674eb308cbab3247f57" gracePeriod=45
Sep 17 18:22:53.747314 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:53.747278    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl" podUID="5caf49c5-c902-4acd-949b-753c244a55c9" containerName="main" containerID="cri-o://b08b954c65eb92a1609a3f489aef672283ad4b4621288fafafc0a34cccdf9c76" gracePeriod=45
Sep 17 18:22:54.210106 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.210059    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl" podUID="5caf49c5-c902-4acd-949b-753c244a55c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 18:22:54.500281 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.500254    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"
Sep 17 18:22:54.581741 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.581706    2581 generic.go:356] "Generic (PLEG): container finished" podID="5caf49c5-c902-4acd-949b-753c244a55c9" containerID="b08b954c65eb92a1609a3f489aef672283ad4b4621288fafafc0a34cccdf9c76" exitCode=0
Sep 17 18:22:54.581928 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.581751    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl" event={"ID":"5caf49c5-c902-4acd-949b-753c244a55c9","Type":"ContainerDied","Data":"b08b954c65eb92a1609a3f489aef672283ad4b4621288fafafc0a34cccdf9c76"}
Sep 17 18:22:54.581928 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.581763    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"
Sep 17 18:22:54.581928 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.581793    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl" event={"ID":"5caf49c5-c902-4acd-949b-753c244a55c9","Type":"ContainerDied","Data":"f504c95918153079943e2b75d1b89e36ee56a466bb4f78feaae41b004ac8b0c2"}
Sep 17 18:22:54.581928 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.581816    2581 scope.go:117] "RemoveContainer" containerID="b08b954c65eb92a1609a3f489aef672283ad4b4621288fafafc0a34cccdf9c76"
Sep 17 18:22:54.584050 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.584030    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5caf49c5-c902-4acd-949b-753c244a55c9-kserve-provision-location\") pod \"5caf49c5-c902-4acd-949b-753c244a55c9\" (UID: \"5caf49c5-c902-4acd-949b-753c244a55c9\") "
Sep 17 18:22:54.584158 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.584084    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5caf49c5-c902-4acd-949b-753c244a55c9-model-cache\") pod \"5caf49c5-c902-4acd-949b-753c244a55c9\" (UID: \"5caf49c5-c902-4acd-949b-753c244a55c9\") "
Sep 17 18:22:54.584158 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.584106    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5caf49c5-c902-4acd-949b-753c244a55c9-tls-certs\") pod \"5caf49c5-c902-4acd-949b-753c244a55c9\" (UID: \"5caf49c5-c902-4acd-949b-753c244a55c9\") "
Sep 17 18:22:54.584158 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.584124    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5caf49c5-c902-4acd-949b-753c244a55c9-home\") pod \"5caf49c5-c902-4acd-949b-753c244a55c9\" (UID: \"5caf49c5-c902-4acd-949b-753c244a55c9\") "
Sep 17 18:22:54.584329 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.584160    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5caf49c5-c902-4acd-949b-753c244a55c9-tmp-dir\") pod \"5caf49c5-c902-4acd-949b-753c244a55c9\" (UID: \"5caf49c5-c902-4acd-949b-753c244a55c9\") "
Sep 17 18:22:54.584329 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.584274    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5caf49c5-c902-4acd-949b-753c244a55c9-dshm\") pod \"5caf49c5-c902-4acd-949b-753c244a55c9\" (UID: \"5caf49c5-c902-4acd-949b-753c244a55c9\") "
Sep 17 18:22:54.584329 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.584325    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8b7f6\" (UniqueName: \"kubernetes.io/projected/5caf49c5-c902-4acd-949b-753c244a55c9-kube-api-access-8b7f6\") pod \"5caf49c5-c902-4acd-949b-753c244a55c9\" (UID: \"5caf49c5-c902-4acd-949b-753c244a55c9\") "
Sep 17 18:22:54.584484 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.584365    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5caf49c5-c902-4acd-949b-753c244a55c9-model-cache" (OuterVolumeSpecName: "model-cache") pod "5caf49c5-c902-4acd-949b-753c244a55c9" (UID: "5caf49c5-c902-4acd-949b-753c244a55c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:54.584484 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.584380    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5caf49c5-c902-4acd-949b-753c244a55c9-cabundle-cert\") pod \"5caf49c5-c902-4acd-949b-753c244a55c9\" (UID: \"5caf49c5-c902-4acd-949b-753c244a55c9\") "
Sep 17 18:22:54.585160 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.584791    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5caf49c5-c902-4acd-949b-753c244a55c9-model-cache\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:22:54.585160 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.584822    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5caf49c5-c902-4acd-949b-753c244a55c9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5caf49c5-c902-4acd-949b-753c244a55c9" (UID: "5caf49c5-c902-4acd-949b-753c244a55c9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:22:54.585160 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.584878    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5caf49c5-c902-4acd-949b-753c244a55c9-home" (OuterVolumeSpecName: "home") pod "5caf49c5-c902-4acd-949b-753c244a55c9" (UID: "5caf49c5-c902-4acd-949b-753c244a55c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:54.586824 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.586793    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5caf49c5-c902-4acd-949b-753c244a55c9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5caf49c5-c902-4acd-949b-753c244a55c9" (UID: "5caf49c5-c902-4acd-949b-753c244a55c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:22:54.587071 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.586837    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5caf49c5-c902-4acd-949b-753c244a55c9-dshm" (OuterVolumeSpecName: "dshm") pod "5caf49c5-c902-4acd-949b-753c244a55c9" (UID: "5caf49c5-c902-4acd-949b-753c244a55c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:54.587071 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.587051    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5caf49c5-c902-4acd-949b-753c244a55c9-kube-api-access-8b7f6" (OuterVolumeSpecName: "kube-api-access-8b7f6") pod "5caf49c5-c902-4acd-949b-753c244a55c9" (UID: "5caf49c5-c902-4acd-949b-753c244a55c9"). InnerVolumeSpecName "kube-api-access-8b7f6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:22:54.597201 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.597174    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5caf49c5-c902-4acd-949b-753c244a55c9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5caf49c5-c902-4acd-949b-753c244a55c9" (UID: "5caf49c5-c902-4acd-949b-753c244a55c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:54.612621 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.612601    2581 scope.go:117] "RemoveContainer" containerID="28d46cec8fc2a518dc88305633527d7a567841abf6d650fbff72fff17fba88d1"
Sep 17 18:22:54.613588 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.613561    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5caf49c5-c902-4acd-949b-753c244a55c9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5caf49c5-c902-4acd-949b-753c244a55c9" (UID: "5caf49c5-c902-4acd-949b-753c244a55c9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:54.620783 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.620762    2581 scope.go:117] "RemoveContainer" containerID="b08b954c65eb92a1609a3f489aef672283ad4b4621288fafafc0a34cccdf9c76"
Sep 17 18:22:54.621084 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:22:54.621065    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b08b954c65eb92a1609a3f489aef672283ad4b4621288fafafc0a34cccdf9c76\": container with ID starting with b08b954c65eb92a1609a3f489aef672283ad4b4621288fafafc0a34cccdf9c76 not found: ID does not exist" containerID="b08b954c65eb92a1609a3f489aef672283ad4b4621288fafafc0a34cccdf9c76"
Sep 17 18:22:54.621137 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.621096    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b08b954c65eb92a1609a3f489aef672283ad4b4621288fafafc0a34cccdf9c76"} err="failed to get container status \"b08b954c65eb92a1609a3f489aef672283ad4b4621288fafafc0a34cccdf9c76\": rpc error: code = NotFound desc = could not find container \"b08b954c65eb92a1609a3f489aef672283ad4b4621288fafafc0a34cccdf9c76\": container with ID starting with b08b954c65eb92a1609a3f489aef672283ad4b4621288fafafc0a34cccdf9c76 not found: ID does not exist"
Sep 17 18:22:54.621137 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.621115    2581 scope.go:117] "RemoveContainer" containerID="28d46cec8fc2a518dc88305633527d7a567841abf6d650fbff72fff17fba88d1"
Sep 17 18:22:54.621427 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:22:54.621407    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28d46cec8fc2a518dc88305633527d7a567841abf6d650fbff72fff17fba88d1\": container with ID starting with 28d46cec8fc2a518dc88305633527d7a567841abf6d650fbff72fff17fba88d1 not found: ID does not exist" containerID="28d46cec8fc2a518dc88305633527d7a567841abf6d650fbff72fff17fba88d1"
Sep 17 18:22:54.621473 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.621432    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28d46cec8fc2a518dc88305633527d7a567841abf6d650fbff72fff17fba88d1"} err="failed to get container status \"28d46cec8fc2a518dc88305633527d7a567841abf6d650fbff72fff17fba88d1\": rpc error: code = NotFound desc = could not find container \"28d46cec8fc2a518dc88305633527d7a567841abf6d650fbff72fff17fba88d1\": container with ID starting with 28d46cec8fc2a518dc88305633527d7a567841abf6d650fbff72fff17fba88d1 not found: ID does not exist"
Sep 17 18:22:54.685888 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.685850    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5caf49c5-c902-4acd-949b-753c244a55c9-kserve-provision-location\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:22:54.685888 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.685885    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5caf49c5-c902-4acd-949b-753c244a55c9-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:22:54.685888 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.685927    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5caf49c5-c902-4acd-949b-753c244a55c9-home\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:22:54.686150 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.685937    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5caf49c5-c902-4acd-949b-753c244a55c9-tmp-dir\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:22:54.686150 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.685947    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5caf49c5-c902-4acd-949b-753c244a55c9-dshm\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:22:54.686150 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.685954    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b7f6\" (UniqueName: \"kubernetes.io/projected/5caf49c5-c902-4acd-949b-753c244a55c9-kube-api-access-8b7f6\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:22:54.686150 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.685962    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5caf49c5-c902-4acd-949b-753c244a55c9-cabundle-cert\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:22:54.906789 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.906703    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"]
Sep 17 18:22:54.908641 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:54.908615    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-849f576695-69lfl"]
Sep 17 18:22:56.359002 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:56.358965    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5caf49c5-c902-4acd-949b-753c244a55c9" path="/var/lib/kubelet/pods/5caf49c5-c902-4acd-949b-753c244a55c9/volumes"
Sep 17 18:22:57.245676 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:57.245631    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7" podUID="2fc77015-7cba-4972-bf20-56aa16bbec21" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:22:59.681064 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:59.681020    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg" podUID="552d1695-23ad-4e98-84b5-e743ee6ac9ee" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:22:59.681491 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:22:59.681150    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg"
Sep 17 18:23:06.640670 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:06.640638    2581 generic.go:356] "Generic (PLEG): container finished" podID="2fc77015-7cba-4972-bf20-56aa16bbec21" containerID="449886dc912917f9ef9a08aafbafb32fb083d5806b6bba53a6d6af6a48675bd1" exitCode=0
Sep 17 18:23:06.641065 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:06.640711    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7" event={"ID":"2fc77015-7cba-4972-bf20-56aa16bbec21","Type":"ContainerDied","Data":"449886dc912917f9ef9a08aafbafb32fb083d5806b6bba53a6d6af6a48675bd1"}
Sep 17 18:23:06.674562 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:06.674536    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7"
Sep 17 18:23:06.791987 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:06.791955    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fc77015-7cba-4972-bf20-56aa16bbec21-tls-certs\") pod \"2fc77015-7cba-4972-bf20-56aa16bbec21\" (UID: \"2fc77015-7cba-4972-bf20-56aa16bbec21\") "
Sep 17 18:23:06.792173 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:06.792039    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2r7gd\" (UniqueName: \"kubernetes.io/projected/2fc77015-7cba-4972-bf20-56aa16bbec21-kube-api-access-2r7gd\") pod \"2fc77015-7cba-4972-bf20-56aa16bbec21\" (UID: \"2fc77015-7cba-4972-bf20-56aa16bbec21\") "
Sep 17 18:23:06.794103 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:06.794074    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fc77015-7cba-4972-bf20-56aa16bbec21-kube-api-access-2r7gd" (OuterVolumeSpecName: "kube-api-access-2r7gd") pod "2fc77015-7cba-4972-bf20-56aa16bbec21" (UID: "2fc77015-7cba-4972-bf20-56aa16bbec21"). InnerVolumeSpecName "kube-api-access-2r7gd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:23:06.794214 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:06.794111    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fc77015-7cba-4972-bf20-56aa16bbec21-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2fc77015-7cba-4972-bf20-56aa16bbec21" (UID: "2fc77015-7cba-4972-bf20-56aa16bbec21"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:23:06.892893 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:06.892854    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r7gd\" (UniqueName: \"kubernetes.io/projected/2fc77015-7cba-4972-bf20-56aa16bbec21-kube-api-access-2r7gd\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:23:06.892893 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:06.892886    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fc77015-7cba-4972-bf20-56aa16bbec21-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:23:07.646854 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:07.646824    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7"
Sep 17 18:23:07.647389 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:07.646862    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7" event={"ID":"2fc77015-7cba-4972-bf20-56aa16bbec21","Type":"ContainerDied","Data":"b117964862cfc6f495d6cab174c84af0e319c57e697b6515f51ac6105447d2b7"}
Sep 17 18:23:07.647389 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:07.646936    2581 scope.go:117] "RemoveContainer" containerID="449886dc912917f9ef9a08aafbafb32fb083d5806b6bba53a6d6af6a48675bd1"
Sep 17 18:23:07.669301 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:07.669272    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7"]
Sep 17 18:23:07.672896 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:07.672869    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785jv9b7"]
Sep 17 18:23:08.358965 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:08.358896    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fc77015-7cba-4972-bf20-56aa16bbec21" path="/var/lib/kubelet/pods/2fc77015-7cba-4972-bf20-56aa16bbec21/volumes"
Sep 17 18:23:09.680596 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:09.680560    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg" podUID="552d1695-23ad-4e98-84b5-e743ee6ac9ee" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:23:19.680385 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:19.680343    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg" podUID="552d1695-23ad-4e98-84b5-e743ee6ac9ee" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:23:21.483672 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.483633    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"]
Sep 17 18:23:21.484074 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.484059    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a" containerName="storage-initializer"
Sep 17 18:23:21.484118 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.484076    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a" containerName="storage-initializer"
Sep 17 18:23:21.484118 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.484083    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a" containerName="main"
Sep 17 18:23:21.484118 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.484089    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a" containerName="main"
Sep 17 18:23:21.484118 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.484097    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5caf49c5-c902-4acd-949b-753c244a55c9" containerName="main"
Sep 17 18:23:21.484118 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.484102    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5caf49c5-c902-4acd-949b-753c244a55c9" containerName="main"
Sep 17 18:23:21.484277 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.484120    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5caf49c5-c902-4acd-949b-753c244a55c9" containerName="storage-initializer"
Sep 17 18:23:21.484277 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.484128    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5caf49c5-c902-4acd-949b-753c244a55c9" containerName="storage-initializer"
Sep 17 18:23:21.484277 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.484140    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fc77015-7cba-4972-bf20-56aa16bbec21" containerName="main"
Sep 17 18:23:21.484277 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.484145    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fc77015-7cba-4972-bf20-56aa16bbec21" containerName="main"
Sep 17 18:23:21.484277 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.484203    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fc77015-7cba-4972-bf20-56aa16bbec21" containerName="main"
Sep 17 18:23:21.484277 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.484236    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d5b6d2a-f96b-478b-9cc5-89fddb97dd6a" containerName="main"
Sep 17 18:23:21.484277 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.484244    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5caf49c5-c902-4acd-949b-753c244a55c9" containerName="main"
Sep 17 18:23:21.487288 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.487271    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"
Sep 17 18:23:21.490082 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.490059    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 17 18:23:21.490082 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.490070    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 17 18:23:21.491127 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.491113    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 17 18:23:21.491202 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.491166    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 17 18:23:21.491259 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.491201    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-9gjr9\""
Sep 17 18:23:21.491259 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.491201    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:23:21.496750 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.496729    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"]
Sep 17 18:23:21.633123 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.633085    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/433371cb-00f0-445d-96b0-bd393955cf40-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-698c9576bb-x6256\" (UID: \"433371cb-00f0-445d-96b0-bd393955cf40\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"
Sep 17 18:23:21.633293 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.633144    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/433371cb-00f0-445d-96b0-bd393955cf40-tls-certs\") pod \"tokenizer-template-args-test-kserve-698c9576bb-x6256\" (UID: \"433371cb-00f0-445d-96b0-bd393955cf40\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"
Sep 17 18:23:21.633293 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.633206    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/433371cb-00f0-445d-96b0-bd393955cf40-tmp-dir\") pod \"tokenizer-template-args-test-kserve-698c9576bb-x6256\" (UID: \"433371cb-00f0-445d-96b0-bd393955cf40\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"
Sep 17 18:23:21.633293 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.633233    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/433371cb-00f0-445d-96b0-bd393955cf40-home\") pod \"tokenizer-template-args-test-kserve-698c9576bb-x6256\" (UID: \"433371cb-00f0-445d-96b0-bd393955cf40\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"
Sep 17 18:23:21.633293 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.633288    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/433371cb-00f0-445d-96b0-bd393955cf40-dshm\") pod \"tokenizer-template-args-test-kserve-698c9576bb-x6256\" (UID: \"433371cb-00f0-445d-96b0-bd393955cf40\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"
Sep 17 18:23:21.633430 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.633310    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcr25\" (UniqueName: \"kubernetes.io/projected/433371cb-00f0-445d-96b0-bd393955cf40-kube-api-access-zcr25\") pod \"tokenizer-template-args-test-kserve-698c9576bb-x6256\" (UID: \"433371cb-00f0-445d-96b0-bd393955cf40\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"
Sep 17 18:23:21.633430 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.633338    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/433371cb-00f0-445d-96b0-bd393955cf40-model-cache\") pod \"tokenizer-template-args-test-kserve-698c9576bb-x6256\" (UID: \"433371cb-00f0-445d-96b0-bd393955cf40\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"
Sep 17 18:23:21.633430 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.633352    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/433371cb-00f0-445d-96b0-bd393955cf40-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-698c9576bb-x6256\" (UID: \"433371cb-00f0-445d-96b0-bd393955cf40\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"
Sep 17 18:23:21.734657 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.734570    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/433371cb-00f0-445d-96b0-bd393955cf40-model-cache\") pod \"tokenizer-template-args-test-kserve-698c9576bb-x6256\" (UID: \"433371cb-00f0-445d-96b0-bd393955cf40\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"
Sep 17 18:23:21.734657 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.734604    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/433371cb-00f0-445d-96b0-bd393955cf40-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-698c9576bb-x6256\" (UID: \"433371cb-00f0-445d-96b0-bd393955cf40\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"
Sep 17 18:23:21.734657 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.734623    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/433371cb-00f0-445d-96b0-bd393955cf40-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-698c9576bb-x6256\" (UID: \"433371cb-00f0-445d-96b0-bd393955cf40\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"
Sep 17 18:23:21.734961 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.734672    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/433371cb-00f0-445d-96b0-bd393955cf40-tls-certs\") pod \"tokenizer-template-args-test-kserve-698c9576bb-x6256\" (UID: \"433371cb-00f0-445d-96b0-bd393955cf40\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"
Sep 17 18:23:21.734961 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.734714    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/433371cb-00f0-445d-96b0-bd393955cf40-tmp-dir\") pod \"tokenizer-template-args-test-kserve-698c9576bb-x6256\" (UID: \"433371cb-00f0-445d-96b0-bd393955cf40\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"
Sep 17 18:23:21.734961 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.734748    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/433371cb-00f0-445d-96b0-bd393955cf40-home\") pod \"tokenizer-template-args-test-kserve-698c9576bb-x6256\" (UID: \"433371cb-00f0-445d-96b0-bd393955cf40\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"
Sep 17 18:23:21.734961 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.734798    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/433371cb-00f0-445d-96b0-bd393955cf40-dshm\") pod \"tokenizer-template-args-test-kserve-698c9576bb-x6256\" (UID: \"433371cb-00f0-445d-96b0-bd393955cf40\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"
Sep 17 18:23:21.734961 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.734831    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcr25\" (UniqueName: \"kubernetes.io/projected/433371cb-00f0-445d-96b0-bd393955cf40-kube-api-access-zcr25\") pod \"tokenizer-template-args-test-kserve-698c9576bb-x6256\" (UID: \"433371cb-00f0-445d-96b0-bd393955cf40\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"
Sep 17 18:23:21.735236 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.735060    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/433371cb-00f0-445d-96b0-bd393955cf40-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-698c9576bb-x6256\" (UID: \"433371cb-00f0-445d-96b0-bd393955cf40\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"
Sep 17 18:23:21.735356 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.735330    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/433371cb-00f0-445d-96b0-bd393955cf40-model-cache\") pod \"tokenizer-template-args-test-kserve-698c9576bb-x6256\" (UID: \"433371cb-00f0-445d-96b0-bd393955cf40\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"
Sep 17 18:23:21.735539 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.735516    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/433371cb-00f0-445d-96b0-bd393955cf40-tmp-dir\") pod \"tokenizer-template-args-test-kserve-698c9576bb-x6256\" (UID: \"433371cb-00f0-445d-96b0-bd393955cf40\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"
Sep 17 18:23:21.735632 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.735581    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/433371cb-00f0-445d-96b0-bd393955cf40-home\") pod \"tokenizer-template-args-test-kserve-698c9576bb-x6256\" (UID: \"433371cb-00f0-445d-96b0-bd393955cf40\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"
Sep 17 18:23:21.735692 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.735658    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/433371cb-00f0-445d-96b0-bd393955cf40-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-698c9576bb-x6256\" (UID: \"433371cb-00f0-445d-96b0-bd393955cf40\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"
Sep 17 18:23:21.737517 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.737488    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/433371cb-00f0-445d-96b0-bd393955cf40-dshm\") pod \"tokenizer-template-args-test-kserve-698c9576bb-x6256\" (UID: \"433371cb-00f0-445d-96b0-bd393955cf40\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"
Sep 17 18:23:21.737697 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.737676    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/433371cb-00f0-445d-96b0-bd393955cf40-tls-certs\") pod \"tokenizer-template-args-test-kserve-698c9576bb-x6256\" (UID: \"433371cb-00f0-445d-96b0-bd393955cf40\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"
Sep 17 18:23:21.752164 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.752129    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcr25\" (UniqueName: \"kubernetes.io/projected/433371cb-00f0-445d-96b0-bd393955cf40-kube-api-access-zcr25\") pod \"tokenizer-template-args-test-kserve-698c9576bb-x6256\" (UID: \"433371cb-00f0-445d-96b0-bd393955cf40\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"
Sep 17 18:23:21.756652 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.756624    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4"]
Sep 17 18:23:21.759871 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.759852    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4"
Sep 17 18:23:21.762612 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.762587    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-epp-sa-dockercfg-7bwf8\""
Sep 17 18:23:21.770319 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.770293    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4"]
Sep 17 18:23:21.799412 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.799372    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"
Sep 17 18:23:21.936758 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.936718    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmb48\" (UniqueName: \"kubernetes.io/projected/07ec416e-e676-46df-ab17-ede9229a5bf3-kube-api-access-qmb48\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4\" (UID: \"07ec416e-e676-46df-ab17-ede9229a5bf3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4"
Sep 17 18:23:21.936962 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:21.936848    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07ec416e-e676-46df-ab17-ede9229a5bf3-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4\" (UID: \"07ec416e-e676-46df-ab17-ede9229a5bf3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4"
Sep 17 18:23:22.038053 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:22.037956    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07ec416e-e676-46df-ab17-ede9229a5bf3-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4\" (UID: \"07ec416e-e676-46df-ab17-ede9229a5bf3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4"
Sep 17 18:23:22.038053 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:22.038038    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmb48\" (UniqueName: \"kubernetes.io/projected/07ec416e-e676-46df-ab17-ede9229a5bf3-kube-api-access-qmb48\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4\" (UID: \"07ec416e-e676-46df-ab17-ede9229a5bf3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4"
Sep 17 18:23:22.040496 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:22.040474    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07ec416e-e676-46df-ab17-ede9229a5bf3-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4\" (UID: \"07ec416e-e676-46df-ab17-ede9229a5bf3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4"
Sep 17 18:23:22.046653 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:22.046634    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmb48\" (UniqueName: \"kubernetes.io/projected/07ec416e-e676-46df-ab17-ede9229a5bf3-kube-api-access-qmb48\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4\" (UID: \"07ec416e-e676-46df-ab17-ede9229a5bf3\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4"
Sep 17 18:23:22.070646 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:22.070605    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4"
Sep 17 18:23:22.074284 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:22.074257    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"]
Sep 17 18:23:22.076037 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:23:22.076014    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433371cb_00f0_445d_96b0_bd393955cf40.slice/crio-207e151e0ed8e9f362709725e60263f91d4a447f75d67d75e164b47552b84326 WatchSource:0}: Error finding container 207e151e0ed8e9f362709725e60263f91d4a447f75d67d75e164b47552b84326: Status 404 returned error can't find the container with id 207e151e0ed8e9f362709725e60263f91d4a447f75d67d75e164b47552b84326
Sep 17 18:23:22.348508 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:22.348482    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4"]
Sep 17 18:23:22.350526 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:23:22.350485    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07ec416e_e676_46df_ab17_ede9229a5bf3.slice/crio-a062da96884d78a8b9b660f9cf35566d03b8747ee083e723ab5fb745f42290d4 WatchSource:0}: Error finding container a062da96884d78a8b9b660f9cf35566d03b8747ee083e723ab5fb745f42290d4: Status 404 returned error can't find the container with id a062da96884d78a8b9b660f9cf35566d03b8747ee083e723ab5fb745f42290d4
Sep 17 18:23:22.719873 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:22.719832    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4" event={"ID":"07ec416e-e676-46df-ab17-ede9229a5bf3","Type":"ContainerStarted","Data":"8b11b8dd4bb702e38a15e12a7d5fed997614074d0a3af2a17cb114b7b2991d06"}
Sep 17 18:23:22.720364 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:22.719912    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4" event={"ID":"07ec416e-e676-46df-ab17-ede9229a5bf3","Type":"ContainerStarted","Data":"a062da96884d78a8b9b660f9cf35566d03b8747ee083e723ab5fb745f42290d4"}
Sep 17 18:23:22.720364 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:22.720035    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4"
Sep 17 18:23:22.721541 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:22.721514    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256" event={"ID":"433371cb-00f0-445d-96b0-bd393955cf40","Type":"ContainerStarted","Data":"4d47a20ae879ad4765140912ce5732741628b717844339b4f53ae326b242daae"}
Sep 17 18:23:22.721648 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:22.721562    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256" event={"ID":"433371cb-00f0-445d-96b0-bd393955cf40","Type":"ContainerStarted","Data":"207e151e0ed8e9f362709725e60263f91d4a447f75d67d75e164b47552b84326"}
Sep 17 18:23:22.737181 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:22.737128    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4" podStartSLOduration=1.737111363 podStartE2EDuration="1.737111363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:23:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:23:22.735596952 +0000 UTC m=+1590.826218614" watchObservedRunningTime="2026-09-17 18:23:22.737111363 +0000 UTC m=+1590.827733012"
Sep 17 18:23:24.008459 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:24.008348    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg"
Sep 17 18:23:24.160826 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:24.160789    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/552d1695-23ad-4e98-84b5-e743ee6ac9ee-tls-certs\") pod \"552d1695-23ad-4e98-84b5-e743ee6ac9ee\" (UID: \"552d1695-23ad-4e98-84b5-e743ee6ac9ee\") "
Sep 17 18:23:24.161035 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:24.161002    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9xc4\" (UniqueName: \"kubernetes.io/projected/552d1695-23ad-4e98-84b5-e743ee6ac9ee-kube-api-access-b9xc4\") pod \"552d1695-23ad-4e98-84b5-e743ee6ac9ee\" (UID: \"552d1695-23ad-4e98-84b5-e743ee6ac9ee\") "
Sep 17 18:23:24.163240 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:24.163204    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/552d1695-23ad-4e98-84b5-e743ee6ac9ee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "552d1695-23ad-4e98-84b5-e743ee6ac9ee" (UID: "552d1695-23ad-4e98-84b5-e743ee6ac9ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:23:24.163504 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:24.163481    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/552d1695-23ad-4e98-84b5-e743ee6ac9ee-kube-api-access-b9xc4" (OuterVolumeSpecName: "kube-api-access-b9xc4") pod "552d1695-23ad-4e98-84b5-e743ee6ac9ee" (UID: "552d1695-23ad-4e98-84b5-e743ee6ac9ee"). InnerVolumeSpecName "kube-api-access-b9xc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:23:24.262875 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:24.262772    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9xc4\" (UniqueName: \"kubernetes.io/projected/552d1695-23ad-4e98-84b5-e743ee6ac9ee-kube-api-access-b9xc4\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:23:24.262875 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:24.262833    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/552d1695-23ad-4e98-84b5-e743ee6ac9ee-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:23:24.734582 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:24.734550    2581 generic.go:356] "Generic (PLEG): container finished" podID="433371cb-00f0-445d-96b0-bd393955cf40" containerID="4d47a20ae879ad4765140912ce5732741628b717844339b4f53ae326b242daae" exitCode=0
Sep 17 18:23:24.734770 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:24.734622    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256" event={"ID":"433371cb-00f0-445d-96b0-bd393955cf40","Type":"ContainerDied","Data":"4d47a20ae879ad4765140912ce5732741628b717844339b4f53ae326b242daae"}
Sep 17 18:23:24.735984 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:24.735959    2581 generic.go:356] "Generic (PLEG): container finished" podID="552d1695-23ad-4e98-84b5-e743ee6ac9ee" containerID="315f487c9c2f64666d1ea7d54afa843cf60a59a1d8154674eb308cbab3247f57" exitCode=0
Sep 17 18:23:24.736101 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:24.736003    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg"
Sep 17 18:23:24.736101 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:24.736005    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg" event={"ID":"552d1695-23ad-4e98-84b5-e743ee6ac9ee","Type":"ContainerDied","Data":"315f487c9c2f64666d1ea7d54afa843cf60a59a1d8154674eb308cbab3247f57"}
Sep 17 18:23:24.736238 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:24.736146    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg" event={"ID":"552d1695-23ad-4e98-84b5-e743ee6ac9ee","Type":"ContainerDied","Data":"d7831d7ca569eae440ed399ee3446f34626e9dc319959fae42b7ec018eaf6614"}
Sep 17 18:23:24.736238 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:24.736185    2581 scope.go:117] "RemoveContainer" containerID="315f487c9c2f64666d1ea7d54afa843cf60a59a1d8154674eb308cbab3247f57"
Sep 17 18:23:24.751040 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:24.751008    2581 scope.go:117] "RemoveContainer" containerID="315f487c9c2f64666d1ea7d54afa843cf60a59a1d8154674eb308cbab3247f57"
Sep 17 18:23:24.751436 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:23:24.751414    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"315f487c9c2f64666d1ea7d54afa843cf60a59a1d8154674eb308cbab3247f57\": container with ID starting with 315f487c9c2f64666d1ea7d54afa843cf60a59a1d8154674eb308cbab3247f57 not found: ID does not exist" containerID="315f487c9c2f64666d1ea7d54afa843cf60a59a1d8154674eb308cbab3247f57"
Sep 17 18:23:24.751515 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:24.751447    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"315f487c9c2f64666d1ea7d54afa843cf60a59a1d8154674eb308cbab3247f57"} err="failed to get container status \"315f487c9c2f64666d1ea7d54afa843cf60a59a1d8154674eb308cbab3247f57\": rpc error: code = NotFound desc = could not find container \"315f487c9c2f64666d1ea7d54afa843cf60a59a1d8154674eb308cbab3247f57\": container with ID starting with 315f487c9c2f64666d1ea7d54afa843cf60a59a1d8154674eb308cbab3247f57 not found: ID does not exist"
Sep 17 18:23:24.772732 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:24.772701    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg"]
Sep 17 18:23:24.774512 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:24.774488    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9c9m4tg"]
Sep 17 18:23:25.742192 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:25.742159    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256" event={"ID":"433371cb-00f0-445d-96b0-bd393955cf40","Type":"ContainerStarted","Data":"187cc9decdde46bb03e24b5d8907ee5349a0e6c785bce2fe5451b01e50df7c4a"}
Sep 17 18:23:25.763624 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:25.763562    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256" podStartSLOduration=2.278956604 podStartE2EDuration="4.763543052s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.484586448s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:23:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:23:25.761260633 +0000 UTC m=+1593.851882283" watchObservedRunningTime="2026-09-17 18:23:25.763543052 +0000 UTC m=+1593.854164701"
Sep 17 18:23:25.799796 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:25.799756    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"
Sep 17 18:23:26.357684 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:26.357653    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="552d1695-23ad-4e98-84b5-e743ee6ac9ee" path="/var/lib/kubelet/pods/552d1695-23ad-4e98-84b5-e743ee6ac9ee/volumes"
Sep 17 18:23:31.800496 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:31.800452    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"
Sep 17 18:23:31.813287 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:31.813249    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"
Sep 17 18:23:32.790669 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:32.790639    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"
Sep 17 18:23:43.273409 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.273375    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"]
Sep 17 18:23:43.273857 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.273764    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="552d1695-23ad-4e98-84b5-e743ee6ac9ee" containerName="main"
Sep 17 18:23:43.273857 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.273780    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="552d1695-23ad-4e98-84b5-e743ee6ac9ee" containerName="main"
Sep 17 18:23:43.273974 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.273884    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="552d1695-23ad-4e98-84b5-e743ee6ac9ee" containerName="main"
Sep 17 18:23:43.276982 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.276960    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"
Sep 17 18:23:43.279637 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.279613    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:23:43.279759 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.279630    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Sep 17 18:23:43.279824 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.279797    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-svnmx\""
Sep 17 18:23:43.279966 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.279947    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 17 18:23:43.280265 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.280249    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 17 18:23:43.280695 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.280674    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 17 18:23:43.289762 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.289733    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"]
Sep 17 18:23:43.340805 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.340771    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-tmp-dir\") pod \"auth-disabled-test-kserve-f98bb9cfb-c74rk\" (UID: \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"
Sep 17 18:23:43.340805 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.340809    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-kserve-provision-location\") pod \"auth-disabled-test-kserve-f98bb9cfb-c74rk\" (UID: \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"
Sep 17 18:23:43.341087 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.340831    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-dshm\") pod \"auth-disabled-test-kserve-f98bb9cfb-c74rk\" (UID: \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"
Sep 17 18:23:43.341087 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.340919    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-home\") pod \"auth-disabled-test-kserve-f98bb9cfb-c74rk\" (UID: \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"
Sep 17 18:23:43.341087 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.341026    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-model-cache\") pod \"auth-disabled-test-kserve-f98bb9cfb-c74rk\" (UID: \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"
Sep 17 18:23:43.341087 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.341084    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbjdk\" (UniqueName: \"kubernetes.io/projected/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-kube-api-access-qbjdk\") pod \"auth-disabled-test-kserve-f98bb9cfb-c74rk\" (UID: \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"
Sep 17 18:23:43.341245 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.341112    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-tls-certs\") pod \"auth-disabled-test-kserve-f98bb9cfb-c74rk\" (UID: \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"
Sep 17 18:23:43.341245 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.341135    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-cabundle-cert\") pod \"auth-disabled-test-kserve-f98bb9cfb-c74rk\" (UID: \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"
Sep 17 18:23:43.442259 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.442220    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-tmp-dir\") pod \"auth-disabled-test-kserve-f98bb9cfb-c74rk\" (UID: \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"
Sep 17 18:23:43.442259 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.442260    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-kserve-provision-location\") pod \"auth-disabled-test-kserve-f98bb9cfb-c74rk\" (UID: \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"
Sep 17 18:23:43.442521 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.442289    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-dshm\") pod \"auth-disabled-test-kserve-f98bb9cfb-c74rk\" (UID: \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"
Sep 17 18:23:43.442521 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.442329    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-home\") pod \"auth-disabled-test-kserve-f98bb9cfb-c74rk\" (UID: \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"
Sep 17 18:23:43.442521 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.442399    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-model-cache\") pod \"auth-disabled-test-kserve-f98bb9cfb-c74rk\" (UID: \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"
Sep 17 18:23:43.442521 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.442469    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbjdk\" (UniqueName: \"kubernetes.io/projected/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-kube-api-access-qbjdk\") pod \"auth-disabled-test-kserve-f98bb9cfb-c74rk\" (UID: \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"
Sep 17 18:23:43.442521 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.442507    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-tls-certs\") pod \"auth-disabled-test-kserve-f98bb9cfb-c74rk\" (UID: \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"
Sep 17 18:23:43.442790 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.442554    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-cabundle-cert\") pod \"auth-disabled-test-kserve-f98bb9cfb-c74rk\" (UID: \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"
Sep 17 18:23:43.442790 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.442640    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-tmp-dir\") pod \"auth-disabled-test-kserve-f98bb9cfb-c74rk\" (UID: \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"
Sep 17 18:23:43.443018 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.442995    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-kserve-provision-location\") pod \"auth-disabled-test-kserve-f98bb9cfb-c74rk\" (UID: \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"
Sep 17 18:23:43.443105 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.443031    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-home\") pod \"auth-disabled-test-kserve-f98bb9cfb-c74rk\" (UID: \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"
Sep 17 18:23:43.443228 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.443203    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-model-cache\") pod \"auth-disabled-test-kserve-f98bb9cfb-c74rk\" (UID: \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"
Sep 17 18:23:43.443305 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.443259    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-cabundle-cert\") pod \"auth-disabled-test-kserve-f98bb9cfb-c74rk\" (UID: \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"
Sep 17 18:23:43.444666 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.444648    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-dshm\") pod \"auth-disabled-test-kserve-f98bb9cfb-c74rk\" (UID: \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"
Sep 17 18:23:43.444980 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.444954    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-tls-certs\") pod \"auth-disabled-test-kserve-f98bb9cfb-c74rk\" (UID: \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"
Sep 17 18:23:43.451551 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.451530    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbjdk\" (UniqueName: \"kubernetes.io/projected/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-kube-api-access-qbjdk\") pod \"auth-disabled-test-kserve-f98bb9cfb-c74rk\" (UID: \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"
Sep 17 18:23:43.493180 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.493146    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7"]
Sep 17 18:23:43.496549 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.496528    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7"
Sep 17 18:23:43.499441 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.499416    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-s5xzs\""
Sep 17 18:23:43.508333 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.508297    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7"]
Sep 17 18:23:43.543419 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.543317    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sxxb\" (UniqueName: \"kubernetes.io/projected/850fc624-f9b9-4100-ad1d-e0f3c24a5b60-kube-api-access-5sxxb\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7\" (UID: \"850fc624-f9b9-4100-ad1d-e0f3c24a5b60\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7"
Sep 17 18:23:43.543419 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.543368    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/850fc624-f9b9-4100-ad1d-e0f3c24a5b60-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7\" (UID: \"850fc624-f9b9-4100-ad1d-e0f3c24a5b60\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7"
Sep 17 18:23:43.589007 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.588969    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"
Sep 17 18:23:43.644691 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.644645    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sxxb\" (UniqueName: \"kubernetes.io/projected/850fc624-f9b9-4100-ad1d-e0f3c24a5b60-kube-api-access-5sxxb\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7\" (UID: \"850fc624-f9b9-4100-ad1d-e0f3c24a5b60\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7"
Sep 17 18:23:43.644818 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.644711    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/850fc624-f9b9-4100-ad1d-e0f3c24a5b60-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7\" (UID: \"850fc624-f9b9-4100-ad1d-e0f3c24a5b60\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7"
Sep 17 18:23:43.648179 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.648150    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/850fc624-f9b9-4100-ad1d-e0f3c24a5b60-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7\" (UID: \"850fc624-f9b9-4100-ad1d-e0f3c24a5b60\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7"
Sep 17 18:23:43.658420 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.658390    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sxxb\" (UniqueName: \"kubernetes.io/projected/850fc624-f9b9-4100-ad1d-e0f3c24a5b60-kube-api-access-5sxxb\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7\" (UID: \"850fc624-f9b9-4100-ad1d-e0f3c24a5b60\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7"
Sep 17 18:23:43.809002 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.808919    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7"
Sep 17 18:23:43.871406 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:43.871373    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"]
Sep 17 18:23:43.872554 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:23:43.872526    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a914e1a_2944_4e2f_9a3e_fe2431826ddb.slice/crio-a47ddea772665a50f7bcb2f2fe07e5e457b484fbf2d07617b808e8903f759382 WatchSource:0}: Error finding container a47ddea772665a50f7bcb2f2fe07e5e457b484fbf2d07617b808e8903f759382: Status 404 returned error can't find the container with id a47ddea772665a50f7bcb2f2fe07e5e457b484fbf2d07617b808e8903f759382
Sep 17 18:23:44.088352 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:44.088313    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7"]
Sep 17 18:23:44.090109 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:23:44.090079    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod850fc624_f9b9_4100_ad1d_e0f3c24a5b60.slice/crio-dcb2d7a0c00d6f504176cb08f29269c80e0337d45de61d5a9c247e98b1e2223e WatchSource:0}: Error finding container dcb2d7a0c00d6f504176cb08f29269c80e0337d45de61d5a9c247e98b1e2223e: Status 404 returned error can't find the container with id dcb2d7a0c00d6f504176cb08f29269c80e0337d45de61d5a9c247e98b1e2223e
Sep 17 18:23:44.836910 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:44.836866    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk" event={"ID":"7a914e1a-2944-4e2f-9a3e-fe2431826ddb","Type":"ContainerStarted","Data":"fd77eb146e8868758afebf0fb9bace995f0c27c6b12e5f55baf480ae7adf8d98"}
Sep 17 18:23:44.837353 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:44.836959    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk" event={"ID":"7a914e1a-2944-4e2f-9a3e-fe2431826ddb","Type":"ContainerStarted","Data":"a47ddea772665a50f7bcb2f2fe07e5e457b484fbf2d07617b808e8903f759382"}
Sep 17 18:23:44.838597 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:44.838567    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7" event={"ID":"850fc624-f9b9-4100-ad1d-e0f3c24a5b60","Type":"ContainerStarted","Data":"70cadaf8e38c6ac2513ff47c21b443e0eefe8cb2c2bcd7f5d0da0fd53cce0bd6"}
Sep 17 18:23:44.838737 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:44.838621    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7" event={"ID":"850fc624-f9b9-4100-ad1d-e0f3c24a5b60","Type":"ContainerStarted","Data":"dcb2d7a0c00d6f504176cb08f29269c80e0337d45de61d5a9c247e98b1e2223e"}
Sep 17 18:23:44.838737 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:44.838667    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7"
Sep 17 18:23:44.875325 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:44.875267    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7" podStartSLOduration=1.875245974 podStartE2EDuration="1.875245974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:23:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:23:44.874819946 +0000 UTC m=+1612.965441596" watchObservedRunningTime="2026-09-17 18:23:44.875245974 +0000 UTC m=+1612.965867625"
Sep 17 18:23:46.851759 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:46.851723    2581 generic.go:356] "Generic (PLEG): container finished" podID="7a914e1a-2944-4e2f-9a3e-fe2431826ddb" containerID="fd77eb146e8868758afebf0fb9bace995f0c27c6b12e5f55baf480ae7adf8d98" exitCode=0
Sep 17 18:23:46.851759 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:46.851762    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk" event={"ID":"7a914e1a-2944-4e2f-9a3e-fe2431826ddb","Type":"ContainerDied","Data":"fd77eb146e8868758afebf0fb9bace995f0c27c6b12e5f55baf480ae7adf8d98"}
Sep 17 18:23:47.858960 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:47.858924    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk" event={"ID":"7a914e1a-2944-4e2f-9a3e-fe2431826ddb","Type":"ContainerStarted","Data":"4763d11c83f46237cbf17ebfccf39a5c737ad4c87b7ced0511394276c34e4efb"}
Sep 17 18:23:47.883820 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:47.883767    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk" podStartSLOduration=2.4722187 podStartE2EDuration="4.883750645s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.411531945s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:23:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:23:47.883157449 +0000 UTC m=+1615.973779099" watchObservedRunningTime="2026-09-17 18:23:47.883750645 +0000 UTC m=+1615.974372295"
Sep 17 18:23:48.589195 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:48.589162    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"
Sep 17 18:23:53.589461 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:53.589422    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"
Sep 17 18:23:53.591135 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:53.591101    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk" podUID="7a914e1a-2944-4e2f-9a3e-fe2431826ddb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 18:23:53.731329 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:23:53.731299    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4"
Sep 17 18:24:01.997299 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:01.997262    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"]
Sep 17 18:24:02.008130 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.007964    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4"]
Sep 17 18:24:02.019932 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:24:02.019748    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:24:02.019932 ip-10-0-130-28 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:24:02.019932 ip-10-0-130-28 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:24:02.019932 ip-10-0-130-28 kubenswrapper[2581]:  > containerID="187cc9decdde46bb03e24b5d8907ee5349a0e6c785bce2fe5451b01e50df7c4a" cmd=["/bin/sleep","15"]
Sep 17 18:24:02.019932 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.019818    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256" message="" err=<
Sep 17 18:24:02.019932 ip-10-0-130-28 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:24:02.019932 ip-10-0-130-28 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:24:02.019932 ip-10-0-130-28 kubenswrapper[2581]:  >
Sep 17 18:24:02.019932 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:24:02.019835    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:24:02.019932 ip-10-0-130-28 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:24:02.019932 ip-10-0-130-28 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:24:02.019932 ip-10-0-130-28 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256" podUID="433371cb-00f0-445d-96b0-bd393955cf40" containerName="main" containerID="cri-o://187cc9decdde46bb03e24b5d8907ee5349a0e6c785bce2fe5451b01e50df7c4a"
Sep 17 18:24:02.021097 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.019893    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256" podUID="433371cb-00f0-445d-96b0-bd393955cf40" containerName="main" containerID="cri-o://187cc9decdde46bb03e24b5d8907ee5349a0e6c785bce2fe5451b01e50df7c4a" gracePeriod=60
Sep 17 18:24:02.388150 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.388114    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"
Sep 17 18:24:02.533588 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.533553    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcr25\" (UniqueName: \"kubernetes.io/projected/433371cb-00f0-445d-96b0-bd393955cf40-kube-api-access-zcr25\") pod \"433371cb-00f0-445d-96b0-bd393955cf40\" (UID: \"433371cb-00f0-445d-96b0-bd393955cf40\") "
Sep 17 18:24:02.533772 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.533600    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/433371cb-00f0-445d-96b0-bd393955cf40-cabundle-cert\") pod \"433371cb-00f0-445d-96b0-bd393955cf40\" (UID: \"433371cb-00f0-445d-96b0-bd393955cf40\") "
Sep 17 18:24:02.533772 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.533644    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/433371cb-00f0-445d-96b0-bd393955cf40-dshm\") pod \"433371cb-00f0-445d-96b0-bd393955cf40\" (UID: \"433371cb-00f0-445d-96b0-bd393955cf40\") "
Sep 17 18:24:02.533772 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.533679    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/433371cb-00f0-445d-96b0-bd393955cf40-home\") pod \"433371cb-00f0-445d-96b0-bd393955cf40\" (UID: \"433371cb-00f0-445d-96b0-bd393955cf40\") "
Sep 17 18:24:02.533772 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.533709    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/433371cb-00f0-445d-96b0-bd393955cf40-kserve-provision-location\") pod \"433371cb-00f0-445d-96b0-bd393955cf40\" (UID: \"433371cb-00f0-445d-96b0-bd393955cf40\") "
Sep 17 18:24:02.533772 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.533757    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/433371cb-00f0-445d-96b0-bd393955cf40-tls-certs\") pod \"433371cb-00f0-445d-96b0-bd393955cf40\" (UID: \"433371cb-00f0-445d-96b0-bd393955cf40\") "
Sep 17 18:24:02.534113 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.533779    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/433371cb-00f0-445d-96b0-bd393955cf40-model-cache\") pod \"433371cb-00f0-445d-96b0-bd393955cf40\" (UID: \"433371cb-00f0-445d-96b0-bd393955cf40\") "
Sep 17 18:24:02.534113 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.533827    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/433371cb-00f0-445d-96b0-bd393955cf40-tmp-dir\") pod \"433371cb-00f0-445d-96b0-bd393955cf40\" (UID: \"433371cb-00f0-445d-96b0-bd393955cf40\") "
Sep 17 18:24:02.534113 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.533952    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433371cb-00f0-445d-96b0-bd393955cf40-home" (OuterVolumeSpecName: "home") pod "433371cb-00f0-445d-96b0-bd393955cf40" (UID: "433371cb-00f0-445d-96b0-bd393955cf40"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:24:02.534262 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.534175    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433371cb-00f0-445d-96b0-bd393955cf40-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "433371cb-00f0-445d-96b0-bd393955cf40" (UID: "433371cb-00f0-445d-96b0-bd393955cf40"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:24:02.534262 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.534091    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/433371cb-00f0-445d-96b0-bd393955cf40-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "433371cb-00f0-445d-96b0-bd393955cf40" (UID: "433371cb-00f0-445d-96b0-bd393955cf40"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:24:02.534401 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.534321    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/433371cb-00f0-445d-96b0-bd393955cf40-cabundle-cert\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:24:02.534401 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.534339    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/433371cb-00f0-445d-96b0-bd393955cf40-home\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:24:02.534401 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.534353    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/433371cb-00f0-445d-96b0-bd393955cf40-tmp-dir\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:24:02.534564 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.534412    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433371cb-00f0-445d-96b0-bd393955cf40-model-cache" (OuterVolumeSpecName: "model-cache") pod "433371cb-00f0-445d-96b0-bd393955cf40" (UID: "433371cb-00f0-445d-96b0-bd393955cf40"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:24:02.536093 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.536066    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433371cb-00f0-445d-96b0-bd393955cf40-dshm" (OuterVolumeSpecName: "dshm") pod "433371cb-00f0-445d-96b0-bd393955cf40" (UID: "433371cb-00f0-445d-96b0-bd393955cf40"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:24:02.536563 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.536538    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/433371cb-00f0-445d-96b0-bd393955cf40-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "433371cb-00f0-445d-96b0-bd393955cf40" (UID: "433371cb-00f0-445d-96b0-bd393955cf40"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:24:02.536563 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.536548    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/433371cb-00f0-445d-96b0-bd393955cf40-kube-api-access-zcr25" (OuterVolumeSpecName: "kube-api-access-zcr25") pod "433371cb-00f0-445d-96b0-bd393955cf40" (UID: "433371cb-00f0-445d-96b0-bd393955cf40"). InnerVolumeSpecName "kube-api-access-zcr25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:24:02.560457 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.560420    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/433371cb-00f0-445d-96b0-bd393955cf40-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "433371cb-00f0-445d-96b0-bd393955cf40" (UID: "433371cb-00f0-445d-96b0-bd393955cf40"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:24:02.635367 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.635329    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcr25\" (UniqueName: \"kubernetes.io/projected/433371cb-00f0-445d-96b0-bd393955cf40-kube-api-access-zcr25\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:24:02.635367 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.635361    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/433371cb-00f0-445d-96b0-bd393955cf40-dshm\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:24:02.635367 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.635371    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/433371cb-00f0-445d-96b0-bd393955cf40-kserve-provision-location\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:24:02.635605 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.635380    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/433371cb-00f0-445d-96b0-bd393955cf40-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:24:02.635605 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.635388    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/433371cb-00f0-445d-96b0-bd393955cf40-model-cache\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:24:02.939844 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.939745    2581 generic.go:356] "Generic (PLEG): container finished" podID="433371cb-00f0-445d-96b0-bd393955cf40" containerID="187cc9decdde46bb03e24b5d8907ee5349a0e6c785bce2fe5451b01e50df7c4a" exitCode=0
Sep 17 18:24:02.939844 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.939799    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256" event={"ID":"433371cb-00f0-445d-96b0-bd393955cf40","Type":"ContainerDied","Data":"187cc9decdde46bb03e24b5d8907ee5349a0e6c785bce2fe5451b01e50df7c4a"}
Sep 17 18:24:02.940116 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.939858    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256" event={"ID":"433371cb-00f0-445d-96b0-bd393955cf40","Type":"ContainerDied","Data":"207e151e0ed8e9f362709725e60263f91d4a447f75d67d75e164b47552b84326"}
Sep 17 18:24:02.940116 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.939882    2581 scope.go:117] "RemoveContainer" containerID="187cc9decdde46bb03e24b5d8907ee5349a0e6c785bce2fe5451b01e50df7c4a"
Sep 17 18:24:02.940116 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.939808    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"
Sep 17 18:24:02.950353 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.950332    2581 scope.go:117] "RemoveContainer" containerID="4d47a20ae879ad4765140912ce5732741628b717844339b4f53ae326b242daae"
Sep 17 18:24:02.959941 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.959895    2581 scope.go:117] "RemoveContainer" containerID="187cc9decdde46bb03e24b5d8907ee5349a0e6c785bce2fe5451b01e50df7c4a"
Sep 17 18:24:02.960822 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:24:02.960515    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"187cc9decdde46bb03e24b5d8907ee5349a0e6c785bce2fe5451b01e50df7c4a\": container with ID starting with 187cc9decdde46bb03e24b5d8907ee5349a0e6c785bce2fe5451b01e50df7c4a not found: ID does not exist" containerID="187cc9decdde46bb03e24b5d8907ee5349a0e6c785bce2fe5451b01e50df7c4a"
Sep 17 18:24:02.960822 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.960557    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"187cc9decdde46bb03e24b5d8907ee5349a0e6c785bce2fe5451b01e50df7c4a"} err="failed to get container status \"187cc9decdde46bb03e24b5d8907ee5349a0e6c785bce2fe5451b01e50df7c4a\": rpc error: code = NotFound desc = could not find container \"187cc9decdde46bb03e24b5d8907ee5349a0e6c785bce2fe5451b01e50df7c4a\": container with ID starting with 187cc9decdde46bb03e24b5d8907ee5349a0e6c785bce2fe5451b01e50df7c4a not found: ID does not exist"
Sep 17 18:24:02.960822 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.960580    2581 scope.go:117] "RemoveContainer" containerID="4d47a20ae879ad4765140912ce5732741628b717844339b4f53ae326b242daae"
Sep 17 18:24:02.961540 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:24:02.960863    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d47a20ae879ad4765140912ce5732741628b717844339b4f53ae326b242daae\": container with ID starting with 4d47a20ae879ad4765140912ce5732741628b717844339b4f53ae326b242daae not found: ID does not exist" containerID="4d47a20ae879ad4765140912ce5732741628b717844339b4f53ae326b242daae"
Sep 17 18:24:02.961540 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.960889    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d47a20ae879ad4765140912ce5732741628b717844339b4f53ae326b242daae"} err="failed to get container status \"4d47a20ae879ad4765140912ce5732741628b717844339b4f53ae326b242daae\": rpc error: code = NotFound desc = could not find container \"4d47a20ae879ad4765140912ce5732741628b717844339b4f53ae326b242daae\": container with ID starting with 4d47a20ae879ad4765140912ce5732741628b717844339b4f53ae326b242daae not found: ID does not exist"
Sep 17 18:24:02.963235 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.963201    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"]
Sep 17 18:24:02.966663 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:02.966637    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-698c9576bb-x6256"]
Sep 17 18:24:03.589741 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:03.589701    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk" podUID="7a914e1a-2944-4e2f-9a3e-fe2431826ddb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 18:24:03.731052 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:03.731009    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4" podUID="07ec416e-e676-46df-ab17-ede9229a5bf3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:24:04.358556 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:04.358516    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="433371cb-00f0-445d-96b0-bd393955cf40" path="/var/lib/kubelet/pods/433371cb-00f0-445d-96b0-bd393955cf40/volumes"
Sep 17 18:24:13.589938 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:13.589879    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk" podUID="7a914e1a-2944-4e2f-9a3e-fe2431826ddb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 18:24:13.730357 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:13.730308    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4" podUID="07ec416e-e676-46df-ab17-ede9229a5bf3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:24:15.847371 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:15.847341    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7"
Sep 17 18:24:17.044491 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:17.044423    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4" podUID="07ec416e-e676-46df-ab17-ede9229a5bf3" containerName="main" containerID="cri-o://8b11b8dd4bb702e38a15e12a7d5fed997614074d0a3af2a17cb114b7b2991d06" gracePeriod=45
Sep 17 18:24:23.589759 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:23.589684    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk" podUID="7a914e1a-2944-4e2f-9a3e-fe2431826ddb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 18:24:23.730563 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:23.730515    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4" podUID="07ec416e-e676-46df-ab17-ede9229a5bf3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:24:23.730719 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:23.730656    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4"
Sep 17 18:24:33.590221 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:33.590169    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk" podUID="7a914e1a-2944-4e2f-9a3e-fe2431826ddb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 18:24:33.731004 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:33.730963    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4" podUID="07ec416e-e676-46df-ab17-ede9229a5bf3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:24:43.590450 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:43.590393    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk" podUID="7a914e1a-2944-4e2f-9a3e-fe2431826ddb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 18:24:43.731506 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:43.731460    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4" podUID="07ec416e-e676-46df-ab17-ede9229a5bf3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:24:47.167065 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:47.167025    2581 generic.go:356] "Generic (PLEG): container finished" podID="07ec416e-e676-46df-ab17-ede9229a5bf3" containerID="8b11b8dd4bb702e38a15e12a7d5fed997614074d0a3af2a17cb114b7b2991d06" exitCode=0
Sep 17 18:24:47.167421 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:47.167092    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4" event={"ID":"07ec416e-e676-46df-ab17-ede9229a5bf3","Type":"ContainerDied","Data":"8b11b8dd4bb702e38a15e12a7d5fed997614074d0a3af2a17cb114b7b2991d06"}
Sep 17 18:24:47.293664 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:47.293637    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4"
Sep 17 18:24:47.364112 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:47.364004    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmb48\" (UniqueName: \"kubernetes.io/projected/07ec416e-e676-46df-ab17-ede9229a5bf3-kube-api-access-qmb48\") pod \"07ec416e-e676-46df-ab17-ede9229a5bf3\" (UID: \"07ec416e-e676-46df-ab17-ede9229a5bf3\") "
Sep 17 18:24:47.364112 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:47.364122    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07ec416e-e676-46df-ab17-ede9229a5bf3-tls-certs\") pod \"07ec416e-e676-46df-ab17-ede9229a5bf3\" (UID: \"07ec416e-e676-46df-ab17-ede9229a5bf3\") "
Sep 17 18:24:47.366211 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:47.366182    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07ec416e-e676-46df-ab17-ede9229a5bf3-kube-api-access-qmb48" (OuterVolumeSpecName: "kube-api-access-qmb48") pod "07ec416e-e676-46df-ab17-ede9229a5bf3" (UID: "07ec416e-e676-46df-ab17-ede9229a5bf3"). InnerVolumeSpecName "kube-api-access-qmb48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:24:47.366328 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:47.366223    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07ec416e-e676-46df-ab17-ede9229a5bf3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "07ec416e-e676-46df-ab17-ede9229a5bf3" (UID: "07ec416e-e676-46df-ab17-ede9229a5bf3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:24:47.464927 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:47.464876    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmb48\" (UniqueName: \"kubernetes.io/projected/07ec416e-e676-46df-ab17-ede9229a5bf3-kube-api-access-qmb48\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:24:47.464927 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:47.464923    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07ec416e-e676-46df-ab17-ede9229a5bf3-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:24:48.174160 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:48.174125    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4"
Sep 17 18:24:48.174160 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:48.174157    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4" event={"ID":"07ec416e-e676-46df-ab17-ede9229a5bf3","Type":"ContainerDied","Data":"a062da96884d78a8b9b660f9cf35566d03b8747ee083e723ab5fb745f42290d4"}
Sep 17 18:24:48.174708 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:48.174212    2581 scope.go:117] "RemoveContainer" containerID="8b11b8dd4bb702e38a15e12a7d5fed997614074d0a3af2a17cb114b7b2991d06"
Sep 17 18:24:48.195764 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:48.195730    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4"]
Sep 17 18:24:48.200542 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:48.200514    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dsj6r4"]
Sep 17 18:24:48.360955 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:48.360915    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07ec416e-e676-46df-ab17-ede9229a5bf3" path="/var/lib/kubelet/pods/07ec416e-e676-46df-ab17-ede9229a5bf3/volumes"
Sep 17 18:24:53.590108 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:24:53.590064    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk" podUID="7a914e1a-2944-4e2f-9a3e-fe2431826ddb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 18:25:03.589893 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:25:03.589851    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk" podUID="7a914e1a-2944-4e2f-9a3e-fe2431826ddb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 18:25:13.590277 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:25:13.590233    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk" podUID="7a914e1a-2944-4e2f-9a3e-fe2431826ddb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 18:25:23.590015 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:25:23.589920    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk" podUID="7a914e1a-2944-4e2f-9a3e-fe2431826ddb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 18:25:33.590030 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:25:33.589978    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk" podUID="7a914e1a-2944-4e2f-9a3e-fe2431826ddb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 18:25:43.600097 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:25:43.600065    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"
Sep 17 18:25:43.608467 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:25:43.608435    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"
Sep 17 18:25:48.298214 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:25:48.298179    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"]
Sep 17 18:25:48.303706 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:25:48.303678    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7"]
Sep 17 18:25:55.847009 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:25:55.846961    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7" podUID="850fc624-f9b9-4100-ad1d-e0f3c24a5b60" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:26:03.322347 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:03.322270    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk" podUID="7a914e1a-2944-4e2f-9a3e-fe2431826ddb" containerName="main" containerID="cri-o://4763d11c83f46237cbf17ebfccf39a5c737ad4c87b7ced0511394276c34e4efb" gracePeriod=45
Sep 17 18:26:03.324096 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:03.324064    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7" podUID="850fc624-f9b9-4100-ad1d-e0f3c24a5b60" containerName="main" containerID="cri-o://70cadaf8e38c6ac2513ff47c21b443e0eefe8cb2c2bcd7f5d0da0fd53cce0bd6" gracePeriod=45
Sep 17 18:26:03.601470 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:03.601381    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk" podUID="7a914e1a-2944-4e2f-9a3e-fe2431826ddb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Sep 17 18:26:04.058719 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.058694    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"
Sep 17 18:26:04.150018 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.149924    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-kserve-provision-location\") pod \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\" (UID: \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\") "
Sep 17 18:26:04.150202 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.150064    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-tmp-dir\") pod \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\" (UID: \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\") "
Sep 17 18:26:04.150202 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.150092    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-tls-certs\") pod \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\" (UID: \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\") "
Sep 17 18:26:04.150202 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.150108    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-cabundle-cert\") pod \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\" (UID: \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\") "
Sep 17 18:26:04.150202 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.150142    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-home\") pod \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\" (UID: \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\") "
Sep 17 18:26:04.150202 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.150159    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-model-cache\") pod \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\" (UID: \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\") "
Sep 17 18:26:04.150202 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.150180    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbjdk\" (UniqueName: \"kubernetes.io/projected/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-kube-api-access-qbjdk\") pod \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\" (UID: \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\") "
Sep 17 18:26:04.150544 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.150223    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-dshm\") pod \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\" (UID: \"7a914e1a-2944-4e2f-9a3e-fe2431826ddb\") "
Sep 17 18:26:04.150544 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.150459    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-model-cache" (OuterVolumeSpecName: "model-cache") pod "7a914e1a-2944-4e2f-9a3e-fe2431826ddb" (UID: "7a914e1a-2944-4e2f-9a3e-fe2431826ddb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:04.150544 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.150521    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7a914e1a-2944-4e2f-9a3e-fe2431826ddb" (UID: "7a914e1a-2944-4e2f-9a3e-fe2431826ddb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:26:04.150704 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.150575    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-model-cache\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:26:04.150841 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.150812    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-home" (OuterVolumeSpecName: "home") pod "7a914e1a-2944-4e2f-9a3e-fe2431826ddb" (UID: "7a914e1a-2944-4e2f-9a3e-fe2431826ddb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:04.153131 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.153098    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-kube-api-access-qbjdk" (OuterVolumeSpecName: "kube-api-access-qbjdk") pod "7a914e1a-2944-4e2f-9a3e-fe2431826ddb" (UID: "7a914e1a-2944-4e2f-9a3e-fe2431826ddb"). InnerVolumeSpecName "kube-api-access-qbjdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:26:04.153234 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.153198    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7a914e1a-2944-4e2f-9a3e-fe2431826ddb" (UID: "7a914e1a-2944-4e2f-9a3e-fe2431826ddb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:26:04.153400 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.153379    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-dshm" (OuterVolumeSpecName: "dshm") pod "7a914e1a-2944-4e2f-9a3e-fe2431826ddb" (UID: "7a914e1a-2944-4e2f-9a3e-fe2431826ddb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:04.162311 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.162286    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7a914e1a-2944-4e2f-9a3e-fe2431826ddb" (UID: "7a914e1a-2944-4e2f-9a3e-fe2431826ddb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:04.173482 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.173450    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7a914e1a-2944-4e2f-9a3e-fe2431826ddb" (UID: "7a914e1a-2944-4e2f-9a3e-fe2431826ddb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:04.251729 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.251694    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-tmp-dir\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:26:04.251729 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.251725    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:26:04.251729 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.251738    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-cabundle-cert\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:26:04.252003 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.251745    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-home\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:26:04.252003 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.251756    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbjdk\" (UniqueName: \"kubernetes.io/projected/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-kube-api-access-qbjdk\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:26:04.252003 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.251765    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-dshm\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:26:04.252003 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.251773    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a914e1a-2944-4e2f-9a3e-fe2431826ddb-kserve-provision-location\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:26:04.572471 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.572382    2581 generic.go:356] "Generic (PLEG): container finished" podID="7a914e1a-2944-4e2f-9a3e-fe2431826ddb" containerID="4763d11c83f46237cbf17ebfccf39a5c737ad4c87b7ced0511394276c34e4efb" exitCode=0
Sep 17 18:26:04.572471 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.572434    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"
Sep 17 18:26:04.572471 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.572442    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk" event={"ID":"7a914e1a-2944-4e2f-9a3e-fe2431826ddb","Type":"ContainerDied","Data":"4763d11c83f46237cbf17ebfccf39a5c737ad4c87b7ced0511394276c34e4efb"}
Sep 17 18:26:04.572980 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.572506    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk" event={"ID":"7a914e1a-2944-4e2f-9a3e-fe2431826ddb","Type":"ContainerDied","Data":"a47ddea772665a50f7bcb2f2fe07e5e457b484fbf2d07617b808e8903f759382"}
Sep 17 18:26:04.572980 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.572530    2581 scope.go:117] "RemoveContainer" containerID="4763d11c83f46237cbf17ebfccf39a5c737ad4c87b7ced0511394276c34e4efb"
Sep 17 18:26:04.581422 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.581402    2581 scope.go:117] "RemoveContainer" containerID="fd77eb146e8868758afebf0fb9bace995f0c27c6b12e5f55baf480ae7adf8d98"
Sep 17 18:26:04.589805 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.589787    2581 scope.go:117] "RemoveContainer" containerID="4763d11c83f46237cbf17ebfccf39a5c737ad4c87b7ced0511394276c34e4efb"
Sep 17 18:26:04.590140 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:26:04.590111    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4763d11c83f46237cbf17ebfccf39a5c737ad4c87b7ced0511394276c34e4efb\": container with ID starting with 4763d11c83f46237cbf17ebfccf39a5c737ad4c87b7ced0511394276c34e4efb not found: ID does not exist" containerID="4763d11c83f46237cbf17ebfccf39a5c737ad4c87b7ced0511394276c34e4efb"
Sep 17 18:26:04.590254 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.590149    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4763d11c83f46237cbf17ebfccf39a5c737ad4c87b7ced0511394276c34e4efb"} err="failed to get container status \"4763d11c83f46237cbf17ebfccf39a5c737ad4c87b7ced0511394276c34e4efb\": rpc error: code = NotFound desc = could not find container \"4763d11c83f46237cbf17ebfccf39a5c737ad4c87b7ced0511394276c34e4efb\": container with ID starting with 4763d11c83f46237cbf17ebfccf39a5c737ad4c87b7ced0511394276c34e4efb not found: ID does not exist"
Sep 17 18:26:04.590254 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.590172    2581 scope.go:117] "RemoveContainer" containerID="fd77eb146e8868758afebf0fb9bace995f0c27c6b12e5f55baf480ae7adf8d98"
Sep 17 18:26:04.590541 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:26:04.590480    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd77eb146e8868758afebf0fb9bace995f0c27c6b12e5f55baf480ae7adf8d98\": container with ID starting with fd77eb146e8868758afebf0fb9bace995f0c27c6b12e5f55baf480ae7adf8d98 not found: ID does not exist" containerID="fd77eb146e8868758afebf0fb9bace995f0c27c6b12e5f55baf480ae7adf8d98"
Sep 17 18:26:04.590541 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.590516    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd77eb146e8868758afebf0fb9bace995f0c27c6b12e5f55baf480ae7adf8d98"} err="failed to get container status \"fd77eb146e8868758afebf0fb9bace995f0c27c6b12e5f55baf480ae7adf8d98\": rpc error: code = NotFound desc = could not find container \"fd77eb146e8868758afebf0fb9bace995f0c27c6b12e5f55baf480ae7adf8d98\": container with ID starting with fd77eb146e8868758afebf0fb9bace995f0c27c6b12e5f55baf480ae7adf8d98 not found: ID does not exist"
Sep 17 18:26:04.591986 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.591964    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"]
Sep 17 18:26:04.595429 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:04.595411    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-f98bb9cfb-c74rk"]
Sep 17 18:26:05.847279 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:05.847241    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7" podUID="850fc624-f9b9-4100-ad1d-e0f3c24a5b60" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:26:06.357607 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:06.357574    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a914e1a-2944-4e2f-9a3e-fe2431826ddb" path="/var/lib/kubelet/pods/7a914e1a-2944-4e2f-9a3e-fe2431826ddb/volumes"
Sep 17 18:26:15.847322 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:15.847280    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7" podUID="850fc624-f9b9-4100-ad1d-e0f3c24a5b60" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:26:15.847862 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:15.847403    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7"
Sep 17 18:26:25.847235 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:25.847190    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7" podUID="850fc624-f9b9-4100-ad1d-e0f3c24a5b60" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:26:33.576547 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:33.576519    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7"
Sep 17 18:26:33.614774 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:33.614682    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/850fc624-f9b9-4100-ad1d-e0f3c24a5b60-tls-certs\") pod \"850fc624-f9b9-4100-ad1d-e0f3c24a5b60\" (UID: \"850fc624-f9b9-4100-ad1d-e0f3c24a5b60\") "
Sep 17 18:26:33.614774 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:33.614729    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sxxb\" (UniqueName: \"kubernetes.io/projected/850fc624-f9b9-4100-ad1d-e0f3c24a5b60-kube-api-access-5sxxb\") pod \"850fc624-f9b9-4100-ad1d-e0f3c24a5b60\" (UID: \"850fc624-f9b9-4100-ad1d-e0f3c24a5b60\") "
Sep 17 18:26:33.616767 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:33.616731    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/850fc624-f9b9-4100-ad1d-e0f3c24a5b60-kube-api-access-5sxxb" (OuterVolumeSpecName: "kube-api-access-5sxxb") pod "850fc624-f9b9-4100-ad1d-e0f3c24a5b60" (UID: "850fc624-f9b9-4100-ad1d-e0f3c24a5b60"). InnerVolumeSpecName "kube-api-access-5sxxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:26:33.616767 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:33.616754    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/850fc624-f9b9-4100-ad1d-e0f3c24a5b60-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "850fc624-f9b9-4100-ad1d-e0f3c24a5b60" (UID: "850fc624-f9b9-4100-ad1d-e0f3c24a5b60"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:26:33.715257 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:33.715217    2581 generic.go:356] "Generic (PLEG): container finished" podID="850fc624-f9b9-4100-ad1d-e0f3c24a5b60" containerID="70cadaf8e38c6ac2513ff47c21b443e0eefe8cb2c2bcd7f5d0da0fd53cce0bd6" exitCode=0
Sep 17 18:26:33.715428 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:33.715267    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7"
Sep 17 18:26:33.715428 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:33.715298    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7" event={"ID":"850fc624-f9b9-4100-ad1d-e0f3c24a5b60","Type":"ContainerDied","Data":"70cadaf8e38c6ac2513ff47c21b443e0eefe8cb2c2bcd7f5d0da0fd53cce0bd6"}
Sep 17 18:26:33.715428 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:33.715240    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/850fc624-f9b9-4100-ad1d-e0f3c24a5b60-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:26:33.715428 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:33.715344    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sxxb\" (UniqueName: \"kubernetes.io/projected/850fc624-f9b9-4100-ad1d-e0f3c24a5b60-kube-api-access-5sxxb\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:26:33.715428 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:33.715353    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7" event={"ID":"850fc624-f9b9-4100-ad1d-e0f3c24a5b60","Type":"ContainerDied","Data":"dcb2d7a0c00d6f504176cb08f29269c80e0337d45de61d5a9c247e98b1e2223e"}
Sep 17 18:26:33.715428 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:33.715380    2581 scope.go:117] "RemoveContainer" containerID="70cadaf8e38c6ac2513ff47c21b443e0eefe8cb2c2bcd7f5d0da0fd53cce0bd6"
Sep 17 18:26:33.724077 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:33.724059    2581 scope.go:117] "RemoveContainer" containerID="70cadaf8e38c6ac2513ff47c21b443e0eefe8cb2c2bcd7f5d0da0fd53cce0bd6"
Sep 17 18:26:33.724376 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:26:33.724353    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70cadaf8e38c6ac2513ff47c21b443e0eefe8cb2c2bcd7f5d0da0fd53cce0bd6\": container with ID starting with 70cadaf8e38c6ac2513ff47c21b443e0eefe8cb2c2bcd7f5d0da0fd53cce0bd6 not found: ID does not exist" containerID="70cadaf8e38c6ac2513ff47c21b443e0eefe8cb2c2bcd7f5d0da0fd53cce0bd6"
Sep 17 18:26:33.724449 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:33.724384    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70cadaf8e38c6ac2513ff47c21b443e0eefe8cb2c2bcd7f5d0da0fd53cce0bd6"} err="failed to get container status \"70cadaf8e38c6ac2513ff47c21b443e0eefe8cb2c2bcd7f5d0da0fd53cce0bd6\": rpc error: code = NotFound desc = could not find container \"70cadaf8e38c6ac2513ff47c21b443e0eefe8cb2c2bcd7f5d0da0fd53cce0bd6\": container with ID starting with 70cadaf8e38c6ac2513ff47c21b443e0eefe8cb2c2bcd7f5d0da0fd53cce0bd6 not found: ID does not exist"
Sep 17 18:26:33.742271 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:33.742242    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7"]
Sep 17 18:26:33.755446 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:33.755421    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-pfjf7"]
Sep 17 18:26:34.357970 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:34.357937    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="850fc624-f9b9-4100-ad1d-e0f3c24a5b60" path="/var/lib/kubelet/pods/850fc624-f9b9-4100-ad1d-e0f3c24a5b60/volumes"
Sep 17 18:26:38.618161 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.618123    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"]
Sep 17 18:26:38.618693 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.618672    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="850fc624-f9b9-4100-ad1d-e0f3c24a5b60" containerName="main"
Sep 17 18:26:38.618745 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.618699    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="850fc624-f9b9-4100-ad1d-e0f3c24a5b60" containerName="main"
Sep 17 18:26:38.618745 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.618717    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="433371cb-00f0-445d-96b0-bd393955cf40" containerName="main"
Sep 17 18:26:38.618745 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.618726    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="433371cb-00f0-445d-96b0-bd393955cf40" containerName="main"
Sep 17 18:26:38.618745 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.618742    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07ec416e-e676-46df-ab17-ede9229a5bf3" containerName="main"
Sep 17 18:26:38.618913 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.618751    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="07ec416e-e676-46df-ab17-ede9229a5bf3" containerName="main"
Sep 17 18:26:38.618913 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.618761    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a914e1a-2944-4e2f-9a3e-fe2431826ddb" containerName="storage-initializer"
Sep 17 18:26:38.618913 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.618770    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a914e1a-2944-4e2f-9a3e-fe2431826ddb" containerName="storage-initializer"
Sep 17 18:26:38.618913 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.618779    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a914e1a-2944-4e2f-9a3e-fe2431826ddb" containerName="main"
Sep 17 18:26:38.618913 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.618789    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a914e1a-2944-4e2f-9a3e-fe2431826ddb" containerName="main"
Sep 17 18:26:38.618913 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.618805    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="433371cb-00f0-445d-96b0-bd393955cf40" containerName="storage-initializer"
Sep 17 18:26:38.618913 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.618814    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="433371cb-00f0-445d-96b0-bd393955cf40" containerName="storage-initializer"
Sep 17 18:26:38.619161 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.618917    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="850fc624-f9b9-4100-ad1d-e0f3c24a5b60" containerName="main"
Sep 17 18:26:38.619161 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.618930    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a914e1a-2944-4e2f-9a3e-fe2431826ddb" containerName="main"
Sep 17 18:26:38.619161 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.618939    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="07ec416e-e676-46df-ab17-ede9229a5bf3" containerName="main"
Sep 17 18:26:38.619161 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.618952    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="433371cb-00f0-445d-96b0-bd393955cf40" containerName="main"
Sep 17 18:26:38.623753 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.623731    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:26:38.626690 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.626667    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Sep 17 18:26:38.626823 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.626759    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 17 18:26:38.626929 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.626828    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-2xx25\""
Sep 17 18:26:38.626929 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.626870    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 17 18:26:38.633131 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.633105    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"]
Sep 17 18:26:38.656965 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.656936    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-8555f5d76c-f4jng\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:26:38.657081 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.656973    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-istio-envoy\") pod \"router-gateway-1-openshift-default-8555f5d76c-f4jng\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:26:38.657081 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.657002    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcmmz\" (UniqueName: \"kubernetes.io/projected/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-kube-api-access-pcmmz\") pod \"router-gateway-1-openshift-default-8555f5d76c-f4jng\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:26:38.657081 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.657051    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-istio-data\") pod \"router-gateway-1-openshift-default-8555f5d76c-f4jng\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:26:38.657081 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.657080    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-workload-certs\") pod \"router-gateway-1-openshift-default-8555f5d76c-f4jng\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:26:38.657236 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.657102    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-workload-socket\") pod \"router-gateway-1-openshift-default-8555f5d76c-f4jng\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:26:38.657236 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.657171    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-istio-podinfo\") pod \"router-gateway-1-openshift-default-8555f5d76c-f4jng\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:26:38.657236 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.657203    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-istio-token\") pod \"router-gateway-1-openshift-default-8555f5d76c-f4jng\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:26:38.657236 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.657229    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-credential-socket\") pod \"router-gateway-1-openshift-default-8555f5d76c-f4jng\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:26:38.758395 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.758362    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcmmz\" (UniqueName: \"kubernetes.io/projected/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-kube-api-access-pcmmz\") pod \"router-gateway-1-openshift-default-8555f5d76c-f4jng\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:26:38.758609 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.758423    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-istio-data\") pod \"router-gateway-1-openshift-default-8555f5d76c-f4jng\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:26:38.758609 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.758468    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-workload-certs\") pod \"router-gateway-1-openshift-default-8555f5d76c-f4jng\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:26:38.758609 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.758500    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-workload-socket\") pod \"router-gateway-1-openshift-default-8555f5d76c-f4jng\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:26:38.758609 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.758532    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-istio-podinfo\") pod \"router-gateway-1-openshift-default-8555f5d76c-f4jng\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:26:38.758609 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.758556    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-istio-token\") pod \"router-gateway-1-openshift-default-8555f5d76c-f4jng\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:26:38.758609 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.758587    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-credential-socket\") pod \"router-gateway-1-openshift-default-8555f5d76c-f4jng\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:26:38.759106 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.758626    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-8555f5d76c-f4jng\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:26:38.759106 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.758656    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-istio-envoy\") pod \"router-gateway-1-openshift-default-8555f5d76c-f4jng\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:26:38.759106 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.758965    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-istio-data\") pod \"router-gateway-1-openshift-default-8555f5d76c-f4jng\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:26:38.759106 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.758987    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-workload-socket\") pod \"router-gateway-1-openshift-default-8555f5d76c-f4jng\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:26:38.759106 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.759094    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-credential-socket\") pod \"router-gateway-1-openshift-default-8555f5d76c-f4jng\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:26:38.759293 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.759153    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-workload-certs\") pod \"router-gateway-1-openshift-default-8555f5d76c-f4jng\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:26:38.759293 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.759278    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-8555f5d76c-f4jng\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:26:38.761412 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.761386    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-istio-envoy\") pod \"router-gateway-1-openshift-default-8555f5d76c-f4jng\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:26:38.761550 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.761532    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-istio-podinfo\") pod \"router-gateway-1-openshift-default-8555f5d76c-f4jng\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:26:38.771765 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.771739    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-istio-token\") pod \"router-gateway-1-openshift-default-8555f5d76c-f4jng\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:26:38.771896 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.771823    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcmmz\" (UniqueName: \"kubernetes.io/projected/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-kube-api-access-pcmmz\") pod \"router-gateway-1-openshift-default-8555f5d76c-f4jng\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:26:38.936399 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:38.936309    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:26:39.210315 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:39.210289    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"]
Sep 17 18:26:39.212944 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:26:39.212920    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c6bff6_413a_410f_bb15_6be4ec9ff4d5.slice/crio-3a49a2b84a1d83d87af224222a47c21686e0836c0a17b49e0186d32317211fd9 WatchSource:0}: Error finding container 3a49a2b84a1d83d87af224222a47c21686e0836c0a17b49e0186d32317211fd9: Status 404 returned error can't find the container with id 3a49a2b84a1d83d87af224222a47c21686e0836c0a17b49e0186d32317211fd9
Sep 17 18:26:39.214892 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:39.214861    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:26:39.215001 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:39.214953    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:26:39.215068 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:39.214998    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:26:39.752355 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:39.752314    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng" event={"ID":"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5","Type":"ContainerStarted","Data":"0ee16edfa167e05cf09e00f460df09b669c9af279ee628a6042c0077db806f04"}
Sep 17 18:26:39.752749 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:39.752396    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng" event={"ID":"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5","Type":"ContainerStarted","Data":"3a49a2b84a1d83d87af224222a47c21686e0836c0a17b49e0186d32317211fd9"}
Sep 17 18:26:39.773326 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:39.773279    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng" podStartSLOduration=1.773265053 podStartE2EDuration="1.773265053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:26:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:26:39.770951039 +0000 UTC m=+1787.861572688" watchObservedRunningTime="2026-09-17 18:26:39.773265053 +0000 UTC m=+1787.863886703"
Sep 17 18:26:39.936548 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:39.936516    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:26:40.941144 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:40.941113    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:26:41.761293 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:41.761257    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:26:41.762202 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:41.762179    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:26:51.764095 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:51.764056    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc"]
Sep 17 18:26:51.768220 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:51.768196    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc"
Sep 17 18:26:51.772202 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:51.772177    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-b9n8s\""
Sep 17 18:26:51.772338 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:51.772177    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 17 18:26:51.772338 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:51.772199    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 17 18:26:51.772338 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:51.772225    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-r7rf5\""
Sep 17 18:26:51.772338 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:51.772264    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 17 18:26:51.778802 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:51.778779    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc"]
Sep 17 18:26:51.875774 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:51.875740    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g22qx\" (UniqueName: \"kubernetes.io/projected/de02dd3c-b1d0-4832-8be1-32873b0ba8cc-kube-api-access-g22qx\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc\" (UID: \"de02dd3c-b1d0-4832-8be1-32873b0ba8cc\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc"
Sep 17 18:26:51.875995 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:51.875810    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de02dd3c-b1d0-4832-8be1-32873b0ba8cc-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc\" (UID: \"de02dd3c-b1d0-4832-8be1-32873b0ba8cc\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc"
Sep 17 18:26:51.976849 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:51.976811    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g22qx\" (UniqueName: \"kubernetes.io/projected/de02dd3c-b1d0-4832-8be1-32873b0ba8cc-kube-api-access-g22qx\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc\" (UID: \"de02dd3c-b1d0-4832-8be1-32873b0ba8cc\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc"
Sep 17 18:26:51.977091 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:51.976985    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de02dd3c-b1d0-4832-8be1-32873b0ba8cc-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc\" (UID: \"de02dd3c-b1d0-4832-8be1-32873b0ba8cc\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc"
Sep 17 18:26:51.979482 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:51.979461    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de02dd3c-b1d0-4832-8be1-32873b0ba8cc-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc\" (UID: \"de02dd3c-b1d0-4832-8be1-32873b0ba8cc\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc"
Sep 17 18:26:51.984580 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:51.984555    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g22qx\" (UniqueName: \"kubernetes.io/projected/de02dd3c-b1d0-4832-8be1-32873b0ba8cc-kube-api-access-g22qx\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc\" (UID: \"de02dd3c-b1d0-4832-8be1-32873b0ba8cc\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc"
Sep 17 18:26:52.081879 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:52.081787    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc"
Sep 17 18:26:52.370674 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:52.370647    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc"]
Sep 17 18:26:52.456494 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:52.456461    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/2.log"
Sep 17 18:26:52.464682 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:52.464651    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/2.log"
Sep 17 18:26:52.819675 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:52.819557    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc" event={"ID":"de02dd3c-b1d0-4832-8be1-32873b0ba8cc","Type":"ContainerStarted","Data":"f4c0939012fb2d5f845af851e9679e3945df4258ed059abeeca95ad8f8d7d61e"}
Sep 17 18:26:52.819675 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:52.819647    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc" event={"ID":"de02dd3c-b1d0-4832-8be1-32873b0ba8cc","Type":"ContainerStarted","Data":"aa7cd79ef50685fc732f955e6cc3775def25b3ea805268a44671bacac6e83d2e"}
Sep 17 18:26:52.820274 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:52.819951    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc"
Sep 17 18:26:52.841230 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:26:52.841163    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc" podStartSLOduration=1.8411397919999999 podStartE2EDuration="1.841139792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:26:52.839265085 +0000 UTC m=+1800.929886748" watchObservedRunningTime="2026-09-17 18:26:52.841139792 +0000 UTC m=+1800.931761443"
Sep 17 18:27:15.674286 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:15.674250    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2"]
Sep 17 18:27:15.677804 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:15.677782    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2"
Sep 17 18:27:15.681257 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:15.681235    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-tn6sl\""
Sep 17 18:27:15.681369 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:15.681235    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Sep 17 18:27:15.688437 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:15.688413    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2"]
Sep 17 18:27:15.702073 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:15.702040    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf55b112-db23-4a6d-b90e-47df70c35aad-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2\" (UID: \"bf55b112-db23-4a6d-b90e-47df70c35aad\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2"
Sep 17 18:27:15.702227 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:15.702096    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf55b112-db23-4a6d-b90e-47df70c35aad-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2\" (UID: \"bf55b112-db23-4a6d-b90e-47df70c35aad\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2"
Sep 17 18:27:15.702227 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:15.702127    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf55b112-db23-4a6d-b90e-47df70c35aad-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2\" (UID: \"bf55b112-db23-4a6d-b90e-47df70c35aad\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2"
Sep 17 18:27:15.702227 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:15.702162    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td5db\" (UniqueName: \"kubernetes.io/projected/bf55b112-db23-4a6d-b90e-47df70c35aad-kube-api-access-td5db\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2\" (UID: \"bf55b112-db23-4a6d-b90e-47df70c35aad\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2"
Sep 17 18:27:15.702405 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:15.702265    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf55b112-db23-4a6d-b90e-47df70c35aad-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2\" (UID: \"bf55b112-db23-4a6d-b90e-47df70c35aad\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2"
Sep 17 18:27:15.702405 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:15.702315    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf55b112-db23-4a6d-b90e-47df70c35aad-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2\" (UID: \"bf55b112-db23-4a6d-b90e-47df70c35aad\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2"
Sep 17 18:27:15.802925 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:15.802864    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf55b112-db23-4a6d-b90e-47df70c35aad-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2\" (UID: \"bf55b112-db23-4a6d-b90e-47df70c35aad\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2"
Sep 17 18:27:15.803121 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:15.802933    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf55b112-db23-4a6d-b90e-47df70c35aad-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2\" (UID: \"bf55b112-db23-4a6d-b90e-47df70c35aad\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2"
Sep 17 18:27:15.803121 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:15.802969    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf55b112-db23-4a6d-b90e-47df70c35aad-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2\" (UID: \"bf55b112-db23-4a6d-b90e-47df70c35aad\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2"
Sep 17 18:27:15.803121 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:15.802993    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf55b112-db23-4a6d-b90e-47df70c35aad-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2\" (UID: \"bf55b112-db23-4a6d-b90e-47df70c35aad\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2"
Sep 17 18:27:15.803121 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:15.803012    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf55b112-db23-4a6d-b90e-47df70c35aad-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2\" (UID: \"bf55b112-db23-4a6d-b90e-47df70c35aad\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2"
Sep 17 18:27:15.803121 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:15.803107    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td5db\" (UniqueName: \"kubernetes.io/projected/bf55b112-db23-4a6d-b90e-47df70c35aad-kube-api-access-td5db\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2\" (UID: \"bf55b112-db23-4a6d-b90e-47df70c35aad\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2"
Sep 17 18:27:15.803467 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:15.803358    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf55b112-db23-4a6d-b90e-47df70c35aad-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2\" (UID: \"bf55b112-db23-4a6d-b90e-47df70c35aad\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2"
Sep 17 18:27:15.803467 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:15.803374    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf55b112-db23-4a6d-b90e-47df70c35aad-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2\" (UID: \"bf55b112-db23-4a6d-b90e-47df70c35aad\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2"
Sep 17 18:27:15.803567 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:15.803458    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf55b112-db23-4a6d-b90e-47df70c35aad-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2\" (UID: \"bf55b112-db23-4a6d-b90e-47df70c35aad\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2"
Sep 17 18:27:15.805283 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:15.805258    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf55b112-db23-4a6d-b90e-47df70c35aad-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2\" (UID: \"bf55b112-db23-4a6d-b90e-47df70c35aad\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2"
Sep 17 18:27:15.805549 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:15.805530    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf55b112-db23-4a6d-b90e-47df70c35aad-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2\" (UID: \"bf55b112-db23-4a6d-b90e-47df70c35aad\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2"
Sep 17 18:27:15.810628 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:15.810609    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td5db\" (UniqueName: \"kubernetes.io/projected/bf55b112-db23-4a6d-b90e-47df70c35aad-kube-api-access-td5db\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2\" (UID: \"bf55b112-db23-4a6d-b90e-47df70c35aad\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2"
Sep 17 18:27:15.988727 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:15.988686    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2"
Sep 17 18:27:16.263530 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:16.263445    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2"]
Sep 17 18:27:16.267540 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:27:16.267505    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf55b112_db23_4a6d_b90e_47df70c35aad.slice/crio-000dcfc2d679406b096d00811876178a76aa1daf774f252fe01cb41b4eefa3e6 WatchSource:0}: Error finding container 000dcfc2d679406b096d00811876178a76aa1daf774f252fe01cb41b4eefa3e6: Status 404 returned error can't find the container with id 000dcfc2d679406b096d00811876178a76aa1daf774f252fe01cb41b4eefa3e6
Sep 17 18:27:16.938744 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:16.938708    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2" event={"ID":"bf55b112-db23-4a6d-b90e-47df70c35aad","Type":"ContainerStarted","Data":"d6cb76fdc22cbb5b74f7f3ac655b74641120efbd1ca86072fc964b1ca591d41f"}
Sep 17 18:27:16.939156 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:16.938780    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2" event={"ID":"bf55b112-db23-4a6d-b90e-47df70c35aad","Type":"ContainerStarted","Data":"000dcfc2d679406b096d00811876178a76aa1daf774f252fe01cb41b4eefa3e6"}
Sep 17 18:27:16.956389 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:16.956323    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2" podStartSLOduration=1.95630216 podStartE2EDuration="1.95630216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:27:16.955364765 +0000 UTC m=+1825.045986416" watchObservedRunningTime="2026-09-17 18:27:16.95630216 +0000 UTC m=+1825.046923811"
Sep 17 18:27:16.989103 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:16.989069    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2"
Sep 17 18:27:23.830342 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:23.830308    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc"
Sep 17 18:27:25.982584 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:25.982548    2581 generic.go:356] "Generic (PLEG): container finished" podID="f3c6bff6-413a-410f-bb15-6be4ec9ff4d5" containerID="0ee16edfa167e05cf09e00f460df09b669c9af279ee628a6042c0077db806f04" exitCode=0
Sep 17 18:27:25.983035 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:25.982622    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng" event={"ID":"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5","Type":"ContainerDied","Data":"0ee16edfa167e05cf09e00f460df09b669c9af279ee628a6042c0077db806f04"}
Sep 17 18:27:25.983097 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:25.983077    2581 scope.go:117] "RemoveContainer" containerID="0ee16edfa167e05cf09e00f460df09b669c9af279ee628a6042c0077db806f04"
Sep 17 18:27:25.984143 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:25.984117    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:27:25.984210 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:25.984203    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:27:25.984258 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:25.984233    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:27:25.989213 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:25.989193    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2"
Sep 17 18:27:26.003533 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:26.003508    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2"
Sep 17 18:27:26.761670 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:26.761634    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:27:26.989011 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:26.988970    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng" event={"ID":"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5","Type":"ContainerStarted","Data":"3155208d4d0df303eb42a5d59485af4419b3c8c6222fd2094f1d43443f4ec8a4"}
Sep 17 18:27:26.999967 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:26.999945    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2"
Sep 17 18:27:27.937273 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:27.937229    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:27:27.938506 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:27.938472    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng" podUID="f3c6bff6-413a-410f-bb15-6be4ec9ff4d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 17 18:27:28.937191 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:28.937150    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng" podUID="f3c6bff6-413a-410f-bb15-6be4ec9ff4d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 17 18:27:29.937019 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:29.936891    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng" podUID="f3c6bff6-413a-410f-bb15-6be4ec9ff4d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 17 18:27:30.937141 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:30.937094    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng" podUID="f3c6bff6-413a-410f-bb15-6be4ec9ff4d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 17 18:27:31.940673 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:31.940642    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:27:31.941172 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:31.940983    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:27:31.941704 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:31.941684    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:27:48.785869 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:48.785832    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq"]
Sep 17 18:27:48.788654 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:48.788633    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq"
Sep 17 18:27:48.792561 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:48.792534    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-dkqpx\""
Sep 17 18:27:48.792668 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:48.792534    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 17 18:27:48.800141 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:48.800115    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq"]
Sep 17 18:27:48.924556 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:48.924519    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19014d1a-e153-4a4f-abe2-0cfcb7b2cc15-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq\" (UID: \"19014d1a-e153-4a4f-abe2-0cfcb7b2cc15\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq"
Sep 17 18:27:48.924739 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:48.924575    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k56p7\" (UniqueName: \"kubernetes.io/projected/19014d1a-e153-4a4f-abe2-0cfcb7b2cc15-kube-api-access-k56p7\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq\" (UID: \"19014d1a-e153-4a4f-abe2-0cfcb7b2cc15\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq"
Sep 17 18:27:49.025422 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:49.025388    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k56p7\" (UniqueName: \"kubernetes.io/projected/19014d1a-e153-4a4f-abe2-0cfcb7b2cc15-kube-api-access-k56p7\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq\" (UID: \"19014d1a-e153-4a4f-abe2-0cfcb7b2cc15\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq"
Sep 17 18:27:49.025594 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:49.025470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19014d1a-e153-4a4f-abe2-0cfcb7b2cc15-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq\" (UID: \"19014d1a-e153-4a4f-abe2-0cfcb7b2cc15\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq"
Sep 17 18:27:49.027879 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:49.027850    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19014d1a-e153-4a4f-abe2-0cfcb7b2cc15-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq\" (UID: \"19014d1a-e153-4a4f-abe2-0cfcb7b2cc15\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq"
Sep 17 18:27:49.033873 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:49.033846    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k56p7\" (UniqueName: \"kubernetes.io/projected/19014d1a-e153-4a4f-abe2-0cfcb7b2cc15-kube-api-access-k56p7\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq\" (UID: \"19014d1a-e153-4a4f-abe2-0cfcb7b2cc15\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq"
Sep 17 18:27:49.100129 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:49.100045    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq"
Sep 17 18:27:49.373358 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:49.373332    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq"]
Sep 17 18:27:49.375654 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:27:49.375619    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19014d1a_e153_4a4f_abe2_0cfcb7b2cc15.slice/crio-4dcabd34c262fe1f8d4eb30e31f34919db596f3176f22e49f426a7611be39a12 WatchSource:0}: Error finding container 4dcabd34c262fe1f8d4eb30e31f34919db596f3176f22e49f426a7611be39a12: Status 404 returned error can't find the container with id 4dcabd34c262fe1f8d4eb30e31f34919db596f3176f22e49f426a7611be39a12
Sep 17 18:27:50.106473 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:50.106442    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq" event={"ID":"19014d1a-e153-4a4f-abe2-0cfcb7b2cc15","Type":"ContainerStarted","Data":"ed32eb91003c786a14bd4036cfbb437a6b63d7bab4a2504536234de76cba4e7c"}
Sep 17 18:27:50.106956 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:50.106511    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq" event={"ID":"19014d1a-e153-4a4f-abe2-0cfcb7b2cc15","Type":"ContainerStarted","Data":"4dcabd34c262fe1f8d4eb30e31f34919db596f3176f22e49f426a7611be39a12"}
Sep 17 18:27:50.106956 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:50.106572    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq"
Sep 17 18:27:50.124701 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:27:50.124649    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq" podStartSLOduration=2.124628797 podStartE2EDuration="2.124628797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:27:50.123351126 +0000 UTC m=+1858.213972778" watchObservedRunningTime="2026-09-17 18:27:50.124628797 +0000 UTC m=+1858.215250449"
Sep 17 18:28:13.819309 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:13.819271    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2"]
Sep 17 18:28:13.828621 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:28:13.828584    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:28:13.828621 ip-10-0-130-28 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:28:13.828621 ip-10-0-130-28 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:28:13.828621 ip-10-0-130-28 kubenswrapper[2581]:  > containerID="d6cb76fdc22cbb5b74f7f3ac655b74641120efbd1ca86072fc964b1ca591d41f" cmd=["/bin/sleep","15"]
Sep 17 18:28:13.828957 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:13.828637    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2" message="" err=<
Sep 17 18:28:13.828957 ip-10-0-130-28 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:28:13.828957 ip-10-0-130-28 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:28:13.828957 ip-10-0-130-28 kubenswrapper[2581]:  >
Sep 17 18:28:13.828957 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:28:13.828653    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:28:13.828957 ip-10-0-130-28 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:28:13.828957 ip-10-0-130-28 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 18:28:13.828957 ip-10-0-130-28 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2" podUID="bf55b112-db23-4a6d-b90e-47df70c35aad" containerName="main" containerID="cri-o://d6cb76fdc22cbb5b74f7f3ac655b74641120efbd1ca86072fc964b1ca591d41f"
Sep 17 18:28:13.828957 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:13.828709    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2" podUID="bf55b112-db23-4a6d-b90e-47df70c35aad" containerName="main" containerID="cri-o://d6cb76fdc22cbb5b74f7f3ac655b74641120efbd1ca86072fc964b1ca591d41f" gracePeriod=60
Sep 17 18:28:14.000448 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:14.000410    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2" podUID="bf55b112-db23-4a6d-b90e-47df70c35aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 17 18:28:14.182363 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:14.182336    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2"
Sep 17 18:28:14.217990 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:14.217952    2581 generic.go:356] "Generic (PLEG): container finished" podID="bf55b112-db23-4a6d-b90e-47df70c35aad" containerID="d6cb76fdc22cbb5b74f7f3ac655b74641120efbd1ca86072fc964b1ca591d41f" exitCode=0
Sep 17 18:28:14.218152 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:14.218003    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2"
Sep 17 18:28:14.218152 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:14.218012    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2" event={"ID":"bf55b112-db23-4a6d-b90e-47df70c35aad","Type":"ContainerDied","Data":"d6cb76fdc22cbb5b74f7f3ac655b74641120efbd1ca86072fc964b1ca591d41f"}
Sep 17 18:28:14.218152 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:14.218061    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2" event={"ID":"bf55b112-db23-4a6d-b90e-47df70c35aad","Type":"ContainerDied","Data":"000dcfc2d679406b096d00811876178a76aa1daf774f252fe01cb41b4eefa3e6"}
Sep 17 18:28:14.218152 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:14.218083    2581 scope.go:117] "RemoveContainer" containerID="d6cb76fdc22cbb5b74f7f3ac655b74641120efbd1ca86072fc964b1ca591d41f"
Sep 17 18:28:14.226955 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:14.226934    2581 scope.go:117] "RemoveContainer" containerID="d6cb76fdc22cbb5b74f7f3ac655b74641120efbd1ca86072fc964b1ca591d41f"
Sep 17 18:28:14.227230 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:28:14.227212    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6cb76fdc22cbb5b74f7f3ac655b74641120efbd1ca86072fc964b1ca591d41f\": container with ID starting with d6cb76fdc22cbb5b74f7f3ac655b74641120efbd1ca86072fc964b1ca591d41f not found: ID does not exist" containerID="d6cb76fdc22cbb5b74f7f3ac655b74641120efbd1ca86072fc964b1ca591d41f"
Sep 17 18:28:14.227289 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:14.227240    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6cb76fdc22cbb5b74f7f3ac655b74641120efbd1ca86072fc964b1ca591d41f"} err="failed to get container status \"d6cb76fdc22cbb5b74f7f3ac655b74641120efbd1ca86072fc964b1ca591d41f\": rpc error: code = NotFound desc = could not find container \"d6cb76fdc22cbb5b74f7f3ac655b74641120efbd1ca86072fc964b1ca591d41f\": container with ID starting with d6cb76fdc22cbb5b74f7f3ac655b74641120efbd1ca86072fc964b1ca591d41f not found: ID does not exist"
Sep 17 18:28:14.263758 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:14.263724    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-td5db\" (UniqueName: \"kubernetes.io/projected/bf55b112-db23-4a6d-b90e-47df70c35aad-kube-api-access-td5db\") pod \"bf55b112-db23-4a6d-b90e-47df70c35aad\" (UID: \"bf55b112-db23-4a6d-b90e-47df70c35aad\") "
Sep 17 18:28:14.263985 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:14.263807    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf55b112-db23-4a6d-b90e-47df70c35aad-model-cache\") pod \"bf55b112-db23-4a6d-b90e-47df70c35aad\" (UID: \"bf55b112-db23-4a6d-b90e-47df70c35aad\") "
Sep 17 18:28:14.263985 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:14.263834    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf55b112-db23-4a6d-b90e-47df70c35aad-tls-certs\") pod \"bf55b112-db23-4a6d-b90e-47df70c35aad\" (UID: \"bf55b112-db23-4a6d-b90e-47df70c35aad\") "
Sep 17 18:28:14.263985 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:14.263854    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf55b112-db23-4a6d-b90e-47df70c35aad-tmp-dir\") pod \"bf55b112-db23-4a6d-b90e-47df70c35aad\" (UID: \"bf55b112-db23-4a6d-b90e-47df70c35aad\") "
Sep 17 18:28:14.263985 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:14.263894    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf55b112-db23-4a6d-b90e-47df70c35aad-dshm\") pod \"bf55b112-db23-4a6d-b90e-47df70c35aad\" (UID: \"bf55b112-db23-4a6d-b90e-47df70c35aad\") "
Sep 17 18:28:14.263985 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:14.263964    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf55b112-db23-4a6d-b90e-47df70c35aad-home\") pod \"bf55b112-db23-4a6d-b90e-47df70c35aad\" (UID: \"bf55b112-db23-4a6d-b90e-47df70c35aad\") "
Sep 17 18:28:14.264303 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:14.264129    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf55b112-db23-4a6d-b90e-47df70c35aad-model-cache" (OuterVolumeSpecName: "model-cache") pod "bf55b112-db23-4a6d-b90e-47df70c35aad" (UID: "bf55b112-db23-4a6d-b90e-47df70c35aad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:28:14.264303 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:14.264140    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf55b112-db23-4a6d-b90e-47df70c35aad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bf55b112-db23-4a6d-b90e-47df70c35aad" (UID: "bf55b112-db23-4a6d-b90e-47df70c35aad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:28:14.264413 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:14.264292    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf55b112-db23-4a6d-b90e-47df70c35aad-home" (OuterVolumeSpecName: "home") pod "bf55b112-db23-4a6d-b90e-47df70c35aad" (UID: "bf55b112-db23-4a6d-b90e-47df70c35aad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:28:14.264413 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:14.264373    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf55b112-db23-4a6d-b90e-47df70c35aad-model-cache\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:28:14.264413 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:14.264393    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf55b112-db23-4a6d-b90e-47df70c35aad-tmp-dir\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:28:14.266237 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:14.266207    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf55b112-db23-4a6d-b90e-47df70c35aad-dshm" (OuterVolumeSpecName: "dshm") pod "bf55b112-db23-4a6d-b90e-47df70c35aad" (UID: "bf55b112-db23-4a6d-b90e-47df70c35aad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:28:14.266237 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:14.266225    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf55b112-db23-4a6d-b90e-47df70c35aad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bf55b112-db23-4a6d-b90e-47df70c35aad" (UID: "bf55b112-db23-4a6d-b90e-47df70c35aad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:28:14.266384 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:14.266263    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf55b112-db23-4a6d-b90e-47df70c35aad-kube-api-access-td5db" (OuterVolumeSpecName: "kube-api-access-td5db") pod "bf55b112-db23-4a6d-b90e-47df70c35aad" (UID: "bf55b112-db23-4a6d-b90e-47df70c35aad"). InnerVolumeSpecName "kube-api-access-td5db". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:28:14.365137 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:14.365052    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td5db\" (UniqueName: \"kubernetes.io/projected/bf55b112-db23-4a6d-b90e-47df70c35aad-kube-api-access-td5db\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:28:14.365137 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:14.365084    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf55b112-db23-4a6d-b90e-47df70c35aad-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:28:14.365137 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:14.365100    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf55b112-db23-4a6d-b90e-47df70c35aad-dshm\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:28:14.365137 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:14.365110    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf55b112-db23-4a6d-b90e-47df70c35aad-home\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:28:14.533993 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:14.533959    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2"]
Sep 17 18:28:14.539242 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:14.539215    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997mbmk2"]
Sep 17 18:28:16.358714 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:16.358634    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf55b112-db23-4a6d-b90e-47df70c35aad" path="/var/lib/kubelet/pods/bf55b112-db23-4a6d-b90e-47df70c35aad/volumes"
Sep 17 18:28:21.050491 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.050453    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"]
Sep 17 18:28:21.050884 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.050648    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng" podUID="f3c6bff6-413a-410f-bb15-6be4ec9ff4d5" containerName="istio-proxy" containerID="cri-o://3155208d4d0df303eb42a5d59485af4419b3c8c6222fd2094f1d43443f4ec8a4" gracePeriod=30
Sep 17 18:28:21.112995 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.112968    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq"
Sep 17 18:28:21.273965 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.273929    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"]
Sep 17 18:28:21.274583 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.274559    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf55b112-db23-4a6d-b90e-47df70c35aad" containerName="main"
Sep 17 18:28:21.274676 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.274584    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf55b112-db23-4a6d-b90e-47df70c35aad" containerName="main"
Sep 17 18:28:21.274710 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.274698    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf55b112-db23-4a6d-b90e-47df70c35aad" containerName="main"
Sep 17 18:28:21.279414 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.279394    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"
Sep 17 18:28:21.282111 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.282087    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Sep 17 18:28:21.282229 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.282124    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 17 18:28:21.282329 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.282296    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 17 18:28:21.282448 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.282416    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:28:21.282515 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.282484    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 17 18:28:21.282649 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.282623    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-srtpr\""
Sep 17 18:28:21.290554 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.290531    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"]
Sep 17 18:28:21.429388 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.429297    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e8ca83e-4fbc-45e5-998f-6df479737ce2-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg\" (UID: \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"
Sep 17 18:28:21.429388 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.429354    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e8ca83e-4fbc-45e5-998f-6df479737ce2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg\" (UID: \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"
Sep 17 18:28:21.429600 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.429403    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e8ca83e-4fbc-45e5-998f-6df479737ce2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg\" (UID: \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"
Sep 17 18:28:21.429600 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.429459    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e8ca83e-4fbc-45e5-998f-6df479737ce2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg\" (UID: \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"
Sep 17 18:28:21.429600 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.429475    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e8ca83e-4fbc-45e5-998f-6df479737ce2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg\" (UID: \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"
Sep 17 18:28:21.429600 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.429543    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9h8v\" (UniqueName: \"kubernetes.io/projected/1e8ca83e-4fbc-45e5-998f-6df479737ce2-kube-api-access-b9h8v\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg\" (UID: \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"
Sep 17 18:28:21.429600 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.429578    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e8ca83e-4fbc-45e5-998f-6df479737ce2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg\" (UID: \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"
Sep 17 18:28:21.429600 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.429597    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e8ca83e-4fbc-45e5-998f-6df479737ce2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg\" (UID: \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"
Sep 17 18:28:21.530780 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.530751    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e8ca83e-4fbc-45e5-998f-6df479737ce2-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg\" (UID: \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"
Sep 17 18:28:21.530961 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.530785    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e8ca83e-4fbc-45e5-998f-6df479737ce2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg\" (UID: \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"
Sep 17 18:28:21.530961 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.530811    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e8ca83e-4fbc-45e5-998f-6df479737ce2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg\" (UID: \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"
Sep 17 18:28:21.530961 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.530840    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e8ca83e-4fbc-45e5-998f-6df479737ce2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg\" (UID: \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"
Sep 17 18:28:21.530961 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.530865    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e8ca83e-4fbc-45e5-998f-6df479737ce2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg\" (UID: \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"
Sep 17 18:28:21.531211 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.530962    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9h8v\" (UniqueName: \"kubernetes.io/projected/1e8ca83e-4fbc-45e5-998f-6df479737ce2-kube-api-access-b9h8v\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg\" (UID: \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"
Sep 17 18:28:21.531211 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.531001    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e8ca83e-4fbc-45e5-998f-6df479737ce2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg\" (UID: \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"
Sep 17 18:28:21.531211 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.531029    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e8ca83e-4fbc-45e5-998f-6df479737ce2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg\" (UID: \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"
Sep 17 18:28:21.531366 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.531248    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e8ca83e-4fbc-45e5-998f-6df479737ce2-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg\" (UID: \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"
Sep 17 18:28:21.531366 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.531269    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e8ca83e-4fbc-45e5-998f-6df479737ce2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg\" (UID: \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"
Sep 17 18:28:21.531446 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.531419    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e8ca83e-4fbc-45e5-998f-6df479737ce2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg\" (UID: \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"
Sep 17 18:28:21.531562 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.531542    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e8ca83e-4fbc-45e5-998f-6df479737ce2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg\" (UID: \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"
Sep 17 18:28:21.531694 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.531676    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e8ca83e-4fbc-45e5-998f-6df479737ce2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg\" (UID: \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"
Sep 17 18:28:21.533709 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.533686    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e8ca83e-4fbc-45e5-998f-6df479737ce2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg\" (UID: \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"
Sep 17 18:28:21.534132 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.534110    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e8ca83e-4fbc-45e5-998f-6df479737ce2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg\" (UID: \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"
Sep 17 18:28:21.540883 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.540862    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9h8v\" (UniqueName: \"kubernetes.io/projected/1e8ca83e-4fbc-45e5-998f-6df479737ce2-kube-api-access-b9h8v\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg\" (UID: \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"
Sep 17 18:28:21.591010 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.590978    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"
Sep 17 18:28:21.868910 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:21.868865    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"]
Sep 17 18:28:21.869733 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:28:21.869710    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e8ca83e_4fbc_45e5_998f_6df479737ce2.slice/crio-c9cb62b7a6eeed7e5a036b2fd4be271a156616087272343dec771cb20bfa4ea4 WatchSource:0}: Error finding container c9cb62b7a6eeed7e5a036b2fd4be271a156616087272343dec771cb20bfa4ea4: Status 404 returned error can't find the container with id c9cb62b7a6eeed7e5a036b2fd4be271a156616087272343dec771cb20bfa4ea4
Sep 17 18:28:22.256793 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:22.256760    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg" event={"ID":"1e8ca83e-4fbc-45e5-998f-6df479737ce2","Type":"ContainerStarted","Data":"fd34503f6764a84e8f09849e3389ce89648cff1747bb70be8902d5a5f69232fe"}
Sep 17 18:28:22.257258 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:22.256809    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg" event={"ID":"1e8ca83e-4fbc-45e5-998f-6df479737ce2","Type":"ContainerStarted","Data":"c9cb62b7a6eeed7e5a036b2fd4be271a156616087272343dec771cb20bfa4ea4"}
Sep 17 18:28:24.269943 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:24.269877    2581 generic.go:356] "Generic (PLEG): container finished" podID="1e8ca83e-4fbc-45e5-998f-6df479737ce2" containerID="fd34503f6764a84e8f09849e3389ce89648cff1747bb70be8902d5a5f69232fe" exitCode=0
Sep 17 18:28:24.270419 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:24.269951    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg" event={"ID":"1e8ca83e-4fbc-45e5-998f-6df479737ce2","Type":"ContainerDied","Data":"fd34503f6764a84e8f09849e3389ce89648cff1747bb70be8902d5a5f69232fe"}
Sep 17 18:28:25.277515 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:25.277474    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg" event={"ID":"1e8ca83e-4fbc-45e5-998f-6df479737ce2","Type":"ContainerStarted","Data":"c75f426f57222f5834609a6e87704ccbf5f3eea7aeee9734f05f1694bc511f49"}
Sep 17 18:28:25.305167 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:25.305110    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg" podStartSLOduration=2.046995498 podStartE2EDuration="4.305095567s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.258100069s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:28:25.303411263 +0000 UTC m=+1893.394032916" watchObservedRunningTime="2026-09-17 18:28:25.305095567 +0000 UTC m=+1893.395717217"
Sep 17 18:28:25.591800 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:25.591715    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"
Sep 17 18:28:26.284791 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:26.284754    2581 generic.go:356] "Generic (PLEG): container finished" podID="f3c6bff6-413a-410f-bb15-6be4ec9ff4d5" containerID="3155208d4d0df303eb42a5d59485af4419b3c8c6222fd2094f1d43443f4ec8a4" exitCode=0
Sep 17 18:28:26.285187 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:26.284819    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng" event={"ID":"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5","Type":"ContainerDied","Data":"3155208d4d0df303eb42a5d59485af4419b3c8c6222fd2094f1d43443f4ec8a4"}
Sep 17 18:28:26.285187 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:26.284932    2581 scope.go:117] "RemoveContainer" containerID="0ee16edfa167e05cf09e00f460df09b669c9af279ee628a6042c0077db806f04"
Sep 17 18:28:26.306956 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:26.306934    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:28:26.379567 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:26.379536    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-workload-socket\") pod \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") "
Sep 17 18:28:26.380133 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:26.379590    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-credential-socket\") pod \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") "
Sep 17 18:28:26.380133 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:26.379645    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-istiod-ca-cert\") pod \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") "
Sep 17 18:28:26.380133 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:26.379663    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-istio-envoy\") pod \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") "
Sep 17 18:28:26.380133 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:26.379686    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pcmmz\" (UniqueName: \"kubernetes.io/projected/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-kube-api-access-pcmmz\") pod \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") "
Sep 17 18:28:26.380133 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:26.379719    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-workload-certs\") pod \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") "
Sep 17 18:28:26.380133 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:26.379742    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-istio-token\") pod \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") "
Sep 17 18:28:26.380133 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:26.379831    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-istio-podinfo\") pod \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") "
Sep 17 18:28:26.380133 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:26.379860    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-istio-data\") pod \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\" (UID: \"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5\") "
Sep 17 18:28:26.380133 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:26.379977    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f3c6bff6-413a-410f-bb15-6be4ec9ff4d5" (UID: "f3c6bff6-413a-410f-bb15-6be4ec9ff4d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:28:26.380700 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:26.380171    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-istio-data" (OuterVolumeSpecName: "istio-data") pod "f3c6bff6-413a-410f-bb15-6be4ec9ff4d5" (UID: "f3c6bff6-413a-410f-bb15-6be4ec9ff4d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:28:26.380700 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:26.380190    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f3c6bff6-413a-410f-bb15-6be4ec9ff4d5" (UID: "f3c6bff6-413a-410f-bb15-6be4ec9ff4d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:28:26.380700 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:26.380289    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f3c6bff6-413a-410f-bb15-6be4ec9ff4d5" (UID: "f3c6bff6-413a-410f-bb15-6be4ec9ff4d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:28:26.380700 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:26.380408    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-istio-data\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:28:26.380700 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:26.380426    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-credential-socket\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:28:26.380700 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:26.380441    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-istiod-ca-cert\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:28:26.380700 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:26.380456    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-workload-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:28:26.381109 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:26.380988    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f3c6bff6-413a-410f-bb15-6be4ec9ff4d5" (UID: "f3c6bff6-413a-410f-bb15-6be4ec9ff4d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:28:26.383072 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:26.383035    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f3c6bff6-413a-410f-bb15-6be4ec9ff4d5" (UID: "f3c6bff6-413a-410f-bb15-6be4ec9ff4d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 18:28:26.383072 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:26.383051    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f3c6bff6-413a-410f-bb15-6be4ec9ff4d5" (UID: "f3c6bff6-413a-410f-bb15-6be4ec9ff4d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:28:26.383271 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:26.383117    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-istio-token" (OuterVolumeSpecName: "istio-token") pod "f3c6bff6-413a-410f-bb15-6be4ec9ff4d5" (UID: "f3c6bff6-413a-410f-bb15-6be4ec9ff4d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:28:26.383271 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:26.383126    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-kube-api-access-pcmmz" (OuterVolumeSpecName: "kube-api-access-pcmmz") pod "f3c6bff6-413a-410f-bb15-6be4ec9ff4d5" (UID: "f3c6bff6-413a-410f-bb15-6be4ec9ff4d5"). InnerVolumeSpecName "kube-api-access-pcmmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:28:26.481286 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:26.481238    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-istio-podinfo\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:28:26.481286 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:26.481270    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-workload-socket\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:28:26.481286 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:26.481278    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-istio-envoy\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:28:26.481286 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:26.481289    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcmmz\" (UniqueName: \"kubernetes.io/projected/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-kube-api-access-pcmmz\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:28:26.481286 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:26.481298    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5-istio-token\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:28:27.291839 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:27.291810    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"
Sep 17 18:28:27.292379 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:27.291806    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng" event={"ID":"f3c6bff6-413a-410f-bb15-6be4ec9ff4d5","Type":"ContainerDied","Data":"3a49a2b84a1d83d87af224222a47c21686e0836c0a17b49e0186d32317211fd9"}
Sep 17 18:28:27.292379 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:27.291957    2581 scope.go:117] "RemoveContainer" containerID="3155208d4d0df303eb42a5d59485af4419b3c8c6222fd2094f1d43443f4ec8a4"
Sep 17 18:28:27.327966 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:27.327929    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"]
Sep 17 18:28:27.329964 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:27.329938    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-8555f5d76c-f4jng"]
Sep 17 18:28:28.359376 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:28.359338    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3c6bff6-413a-410f-bb15-6be4ec9ff4d5" path="/var/lib/kubelet/pods/f3c6bff6-413a-410f-bb15-6be4ec9ff4d5/volumes"
Sep 17 18:28:31.592078 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:31.592043    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"
Sep 17 18:28:31.593802 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:31.593776    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg" podUID="1e8ca83e-4fbc-45e5-998f-6df479737ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 18:28:35.885785 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:35.885738    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq"]
Sep 17 18:28:41.113378 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:41.113327    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq" podUID="19014d1a-e153-4a4f-abe2-0cfcb7b2cc15" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:28:41.591512 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:41.591468    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg" podUID="1e8ca83e-4fbc-45e5-998f-6df479737ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 18:28:50.908733 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:50.908691    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq" podUID="19014d1a-e153-4a4f-abe2-0cfcb7b2cc15" containerName="main" containerID="cri-o://ed32eb91003c786a14bd4036cfbb437a6b63d7bab4a2504536234de76cba4e7c" gracePeriod=45
Sep 17 18:28:51.113425 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:51.113382    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq" podUID="19014d1a-e153-4a4f-abe2-0cfcb7b2cc15" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:28:51.592389 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:28:51.592352    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg" podUID="1e8ca83e-4fbc-45e5-998f-6df479737ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 18:29:01.113432 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:29:01.113393    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq" podUID="19014d1a-e153-4a4f-abe2-0cfcb7b2cc15" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:29:01.113925 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:29:01.113499    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq"
Sep 17 18:29:01.591762 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:29:01.591713    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg" podUID="1e8ca83e-4fbc-45e5-998f-6df479737ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 18:29:11.113270 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:29:11.113230    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq" podUID="19014d1a-e153-4a4f-abe2-0cfcb7b2cc15" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:29:11.591958 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:29:11.591890    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg" podUID="1e8ca83e-4fbc-45e5-998f-6df479737ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 18:29:21.146714 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:29:21.146686    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq"
Sep 17 18:29:21.299177 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:29:21.299137    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k56p7\" (UniqueName: \"kubernetes.io/projected/19014d1a-e153-4a4f-abe2-0cfcb7b2cc15-kube-api-access-k56p7\") pod \"19014d1a-e153-4a4f-abe2-0cfcb7b2cc15\" (UID: \"19014d1a-e153-4a4f-abe2-0cfcb7b2cc15\") "
Sep 17 18:29:21.299388 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:29:21.299194    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19014d1a-e153-4a4f-abe2-0cfcb7b2cc15-tls-certs\") pod \"19014d1a-e153-4a4f-abe2-0cfcb7b2cc15\" (UID: \"19014d1a-e153-4a4f-abe2-0cfcb7b2cc15\") "
Sep 17 18:29:21.301492 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:29:21.301459    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19014d1a-e153-4a4f-abe2-0cfcb7b2cc15-kube-api-access-k56p7" (OuterVolumeSpecName: "kube-api-access-k56p7") pod "19014d1a-e153-4a4f-abe2-0cfcb7b2cc15" (UID: "19014d1a-e153-4a4f-abe2-0cfcb7b2cc15"). InnerVolumeSpecName "kube-api-access-k56p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:29:21.301492 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:29:21.301467    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19014d1a-e153-4a4f-abe2-0cfcb7b2cc15-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "19014d1a-e153-4a4f-abe2-0cfcb7b2cc15" (UID: "19014d1a-e153-4a4f-abe2-0cfcb7b2cc15"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:29:21.400466 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:29:21.400414    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k56p7\" (UniqueName: \"kubernetes.io/projected/19014d1a-e153-4a4f-abe2-0cfcb7b2cc15-kube-api-access-k56p7\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:29:21.400466 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:29:21.400449    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19014d1a-e153-4a4f-abe2-0cfcb7b2cc15-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:29:21.568972 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:29:21.568856    2581 generic.go:356] "Generic (PLEG): container finished" podID="19014d1a-e153-4a4f-abe2-0cfcb7b2cc15" containerID="ed32eb91003c786a14bd4036cfbb437a6b63d7bab4a2504536234de76cba4e7c" exitCode=0
Sep 17 18:29:21.568972 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:29:21.568913    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq"
Sep 17 18:29:21.568972 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:29:21.568945    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq" event={"ID":"19014d1a-e153-4a4f-abe2-0cfcb7b2cc15","Type":"ContainerDied","Data":"ed32eb91003c786a14bd4036cfbb437a6b63d7bab4a2504536234de76cba4e7c"}
Sep 17 18:29:21.569275 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:29:21.568999    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq" event={"ID":"19014d1a-e153-4a4f-abe2-0cfcb7b2cc15","Type":"ContainerDied","Data":"4dcabd34c262fe1f8d4eb30e31f34919db596f3176f22e49f426a7611be39a12"}
Sep 17 18:29:21.569275 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:29:21.569022    2581 scope.go:117] "RemoveContainer" containerID="ed32eb91003c786a14bd4036cfbb437a6b63d7bab4a2504536234de76cba4e7c"
Sep 17 18:29:21.578975 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:29:21.578951    2581 scope.go:117] "RemoveContainer" containerID="ed32eb91003c786a14bd4036cfbb437a6b63d7bab4a2504536234de76cba4e7c"
Sep 17 18:29:21.579280 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:29:21.579257    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed32eb91003c786a14bd4036cfbb437a6b63d7bab4a2504536234de76cba4e7c\": container with ID starting with ed32eb91003c786a14bd4036cfbb437a6b63d7bab4a2504536234de76cba4e7c not found: ID does not exist" containerID="ed32eb91003c786a14bd4036cfbb437a6b63d7bab4a2504536234de76cba4e7c"
Sep 17 18:29:21.579400 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:29:21.579285    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed32eb91003c786a14bd4036cfbb437a6b63d7bab4a2504536234de76cba4e7c"} err="failed to get container status \"ed32eb91003c786a14bd4036cfbb437a6b63d7bab4a2504536234de76cba4e7c\": rpc error: code = NotFound desc = could not find container \"ed32eb91003c786a14bd4036cfbb437a6b63d7bab4a2504536234de76cba4e7c\": container with ID starting with ed32eb91003c786a14bd4036cfbb437a6b63d7bab4a2504536234de76cba4e7c not found: ID does not exist"
Sep 17 18:29:21.591041 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:29:21.591009    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq"]
Sep 17 18:29:21.591345 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:29:21.591321    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg" podUID="1e8ca83e-4fbc-45e5-998f-6df479737ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 18:29:21.594523 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:29:21.594497    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq"]
Sep 17 18:29:22.111700 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:29:22.111657    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche89wsq" podUID="19014d1a-e153-4a4f-abe2-0cfcb7b2cc15" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.133.0.54:9003\" within 1s: context deadline exceeded"
Sep 17 18:29:22.111883 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:29:22.111746    2581 logging.go:55] [core] [Channel #682 SubChannel #683]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.54:9003", ServerName: "10.133.0.54:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.54:9003: operation was canceled"
Sep 17 18:29:22.360215 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:29:22.360179    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19014d1a-e153-4a4f-abe2-0cfcb7b2cc15" path="/var/lib/kubelet/pods/19014d1a-e153-4a4f-abe2-0cfcb7b2cc15/volumes"
Sep 17 18:29:24.795562 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:29:24.795527    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc"]
Sep 17 18:29:31.592144 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:29:31.592098    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg" podUID="1e8ca83e-4fbc-45e5-998f-6df479737ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 18:29:33.829564 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:29:33.829524    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc" podUID="de02dd3c-b1d0-4832-8be1-32873b0ba8cc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:29:39.815192 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:29:39.815144    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc" podUID="de02dd3c-b1d0-4832-8be1-32873b0ba8cc" containerName="main" containerID="cri-o://f4c0939012fb2d5f845af851e9679e3945df4258ed059abeeca95ad8f8d7d61e" gracePeriod=45
Sep 17 18:29:41.591629 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:29:41.591591    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg" podUID="1e8ca83e-4fbc-45e5-998f-6df479737ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 18:29:43.830025 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:29:43.829989    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc" podUID="de02dd3c-b1d0-4832-8be1-32873b0ba8cc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:29:51.591622 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:29:51.591523    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg" podUID="1e8ca83e-4fbc-45e5-998f-6df479737ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 18:29:53.830497 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:29:53.830446    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc" podUID="de02dd3c-b1d0-4832-8be1-32873b0ba8cc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:29:53.830983 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:29:53.830572    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc"
Sep 17 18:30:01.592436 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:01.592390    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg" podUID="1e8ca83e-4fbc-45e5-998f-6df479737ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 18:30:03.829756 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:03.829714    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc" podUID="de02dd3c-b1d0-4832-8be1-32873b0ba8cc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:30:10.065935 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:10.065894    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc"
Sep 17 18:30:10.160538 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:10.160439    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g22qx\" (UniqueName: \"kubernetes.io/projected/de02dd3c-b1d0-4832-8be1-32873b0ba8cc-kube-api-access-g22qx\") pod \"de02dd3c-b1d0-4832-8be1-32873b0ba8cc\" (UID: \"de02dd3c-b1d0-4832-8be1-32873b0ba8cc\") "
Sep 17 18:30:10.160694 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:10.160555    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de02dd3c-b1d0-4832-8be1-32873b0ba8cc-tls-certs\") pod \"de02dd3c-b1d0-4832-8be1-32873b0ba8cc\" (UID: \"de02dd3c-b1d0-4832-8be1-32873b0ba8cc\") "
Sep 17 18:30:10.162534 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:10.162499    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de02dd3c-b1d0-4832-8be1-32873b0ba8cc-kube-api-access-g22qx" (OuterVolumeSpecName: "kube-api-access-g22qx") pod "de02dd3c-b1d0-4832-8be1-32873b0ba8cc" (UID: "de02dd3c-b1d0-4832-8be1-32873b0ba8cc"). InnerVolumeSpecName "kube-api-access-g22qx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:30:10.162534 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:10.162516    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de02dd3c-b1d0-4832-8be1-32873b0ba8cc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "de02dd3c-b1d0-4832-8be1-32873b0ba8cc" (UID: "de02dd3c-b1d0-4832-8be1-32873b0ba8cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:30:10.261996 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:10.261956    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g22qx\" (UniqueName: \"kubernetes.io/projected/de02dd3c-b1d0-4832-8be1-32873b0ba8cc-kube-api-access-g22qx\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:30:10.261996 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:10.261988    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de02dd3c-b1d0-4832-8be1-32873b0ba8cc-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:30:10.814296 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:10.814207    2581 generic.go:356] "Generic (PLEG): container finished" podID="de02dd3c-b1d0-4832-8be1-32873b0ba8cc" containerID="f4c0939012fb2d5f845af851e9679e3945df4258ed059abeeca95ad8f8d7d61e" exitCode=0
Sep 17 18:30:10.814296 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:10.814257    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc"
Sep 17 18:30:10.814509 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:10.814300    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc" event={"ID":"de02dd3c-b1d0-4832-8be1-32873b0ba8cc","Type":"ContainerDied","Data":"f4c0939012fb2d5f845af851e9679e3945df4258ed059abeeca95ad8f8d7d61e"}
Sep 17 18:30:10.814509 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:10.814368    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc" event={"ID":"de02dd3c-b1d0-4832-8be1-32873b0ba8cc","Type":"ContainerDied","Data":"aa7cd79ef50685fc732f955e6cc3775def25b3ea805268a44671bacac6e83d2e"}
Sep 17 18:30:10.814509 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:10.814400    2581 scope.go:117] "RemoveContainer" containerID="f4c0939012fb2d5f845af851e9679e3945df4258ed059abeeca95ad8f8d7d61e"
Sep 17 18:30:10.824479 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:10.824458    2581 scope.go:117] "RemoveContainer" containerID="f4c0939012fb2d5f845af851e9679e3945df4258ed059abeeca95ad8f8d7d61e"
Sep 17 18:30:10.824881 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:30:10.824861    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4c0939012fb2d5f845af851e9679e3945df4258ed059abeeca95ad8f8d7d61e\": container with ID starting with f4c0939012fb2d5f845af851e9679e3945df4258ed059abeeca95ad8f8d7d61e not found: ID does not exist" containerID="f4c0939012fb2d5f845af851e9679e3945df4258ed059abeeca95ad8f8d7d61e"
Sep 17 18:30:10.824967 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:10.824890    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4c0939012fb2d5f845af851e9679e3945df4258ed059abeeca95ad8f8d7d61e"} err="failed to get container status \"f4c0939012fb2d5f845af851e9679e3945df4258ed059abeeca95ad8f8d7d61e\": rpc error: code = NotFound desc = could not find container \"f4c0939012fb2d5f845af851e9679e3945df4258ed059abeeca95ad8f8d7d61e\": container with ID starting with f4c0939012fb2d5f845af851e9679e3945df4258ed059abeeca95ad8f8d7d61e not found: ID does not exist"
Sep 17 18:30:10.831249 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:10.831220    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc"]
Sep 17 18:30:10.835555 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:10.835527    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche9f4zc"]
Sep 17 18:30:11.592417 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:11.592365    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg" podUID="1e8ca83e-4fbc-45e5-998f-6df479737ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 18:30:12.358185 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:12.358152    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de02dd3c-b1d0-4832-8be1-32873b0ba8cc" path="/var/lib/kubelet/pods/de02dd3c-b1d0-4832-8be1-32873b0ba8cc/volumes"
Sep 17 18:30:21.601495 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:21.601465    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"
Sep 17 18:30:21.609493 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:21.609465    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"
Sep 17 18:30:25.986060 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:25.986018    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm"]
Sep 17 18:30:25.986572 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:25.986550    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3c6bff6-413a-410f-bb15-6be4ec9ff4d5" containerName="istio-proxy"
Sep 17 18:30:25.986658 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:25.986574    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3c6bff6-413a-410f-bb15-6be4ec9ff4d5" containerName="istio-proxy"
Sep 17 18:30:25.986658 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:25.986601    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de02dd3c-b1d0-4832-8be1-32873b0ba8cc" containerName="main"
Sep 17 18:30:25.986658 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:25.986609    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="de02dd3c-b1d0-4832-8be1-32873b0ba8cc" containerName="main"
Sep 17 18:30:25.986658 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:25.986626    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19014d1a-e153-4a4f-abe2-0cfcb7b2cc15" containerName="main"
Sep 17 18:30:25.986658 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:25.986635    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="19014d1a-e153-4a4f-abe2-0cfcb7b2cc15" containerName="main"
Sep 17 18:30:25.986924 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:25.986745    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3c6bff6-413a-410f-bb15-6be4ec9ff4d5" containerName="istio-proxy"
Sep 17 18:30:25.986924 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:25.986759    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3c6bff6-413a-410f-bb15-6be4ec9ff4d5" containerName="istio-proxy"
Sep 17 18:30:25.986924 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:25.986770    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="de02dd3c-b1d0-4832-8be1-32873b0ba8cc" containerName="main"
Sep 17 18:30:25.986924 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:25.986781    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="19014d1a-e153-4a4f-abe2-0cfcb7b2cc15" containerName="main"
Sep 17 18:30:25.989744 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:25.989717    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm"
Sep 17 18:30:25.993582 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:25.993553    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 17 18:30:25.993717 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:25.993557    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 17 18:30:25.993717 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:25.993563    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-bpbhb\""
Sep 17 18:30:25.993717 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:25.993614    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-cxtj7\""
Sep 17 18:30:25.993938 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:25.993563    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 17 18:30:25.998675 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:25.998650    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm"]
Sep 17 18:30:26.120024 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:26.119986    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/683cd26d-2b55-404d-b799-1032049756cb-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm\" (UID: \"683cd26d-2b55-404d-b799-1032049756cb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm"
Sep 17 18:30:26.120204 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:26.120079    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dscjw\" (UniqueName: \"kubernetes.io/projected/683cd26d-2b55-404d-b799-1032049756cb-kube-api-access-dscjw\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm\" (UID: \"683cd26d-2b55-404d-b799-1032049756cb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm"
Sep 17 18:30:26.220984 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:26.220945    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dscjw\" (UniqueName: \"kubernetes.io/projected/683cd26d-2b55-404d-b799-1032049756cb-kube-api-access-dscjw\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm\" (UID: \"683cd26d-2b55-404d-b799-1032049756cb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm"
Sep 17 18:30:26.221198 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:26.221007    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/683cd26d-2b55-404d-b799-1032049756cb-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm\" (UID: \"683cd26d-2b55-404d-b799-1032049756cb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm"
Sep 17 18:30:26.223523 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:26.223496    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/683cd26d-2b55-404d-b799-1032049756cb-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm\" (UID: \"683cd26d-2b55-404d-b799-1032049756cb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm"
Sep 17 18:30:26.229425 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:26.229391    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dscjw\" (UniqueName: \"kubernetes.io/projected/683cd26d-2b55-404d-b799-1032049756cb-kube-api-access-dscjw\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm\" (UID: \"683cd26d-2b55-404d-b799-1032049756cb\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm"
Sep 17 18:30:26.302016 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:26.301868    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm"
Sep 17 18:30:26.573985 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:26.573958    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm"]
Sep 17 18:30:26.576799 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:30:26.576773    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod683cd26d_2b55_404d_b799_1032049756cb.slice/crio-48f52f03f0881c5c0fe5c1557f6ac8fb41436189bdc5e9a37c0d96241ba1404f WatchSource:0}: Error finding container 48f52f03f0881c5c0fe5c1557f6ac8fb41436189bdc5e9a37c0d96241ba1404f: Status 404 returned error can't find the container with id 48f52f03f0881c5c0fe5c1557f6ac8fb41436189bdc5e9a37c0d96241ba1404f
Sep 17 18:30:26.894942 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:26.894829    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm" event={"ID":"683cd26d-2b55-404d-b799-1032049756cb","Type":"ContainerStarted","Data":"5d96e644434a9505b459dce57cac6d8adf18668295b654d8bda432ca31c34654"}
Sep 17 18:30:26.894942 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:26.894935    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm"
Sep 17 18:30:26.895128 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:26.894967    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm" event={"ID":"683cd26d-2b55-404d-b799-1032049756cb","Type":"ContainerStarted","Data":"48f52f03f0881c5c0fe5c1557f6ac8fb41436189bdc5e9a37c0d96241ba1404f"}
Sep 17 18:30:26.916761 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:26.916706    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm" podStartSLOduration=1.9166850119999999 podStartE2EDuration="1.916685012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:26.914126994 +0000 UTC m=+2015.004748688" watchObservedRunningTime="2026-09-17 18:30:26.916685012 +0000 UTC m=+2015.007306663"
Sep 17 18:30:57.902310 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:57.902281    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm"
Sep 17 18:30:59.374966 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:30:59.374928    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"]
Sep 17 18:31:10.499162 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:10.499122    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt"]
Sep 17 18:31:10.499640 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:10.499474    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3c6bff6-413a-410f-bb15-6be4ec9ff4d5" containerName="istio-proxy"
Sep 17 18:31:10.499640 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:10.499486    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3c6bff6-413a-410f-bb15-6be4ec9ff4d5" containerName="istio-proxy"
Sep 17 18:31:10.502056 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:10.502033    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt"
Sep 17 18:31:10.506168 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:10.506144    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 17 18:31:10.506288 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:10.506167    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-h2rk9\""
Sep 17 18:31:10.512648 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:10.512620    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt"]
Sep 17 18:31:10.623770 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:10.623738    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10a62bca-19fb-4fc7-a31a-ca2080ae45d7-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt\" (UID: \"10a62bca-19fb-4fc7-a31a-ca2080ae45d7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt"
Sep 17 18:31:10.623770 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:10.623775    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg8qb\" (UniqueName: \"kubernetes.io/projected/10a62bca-19fb-4fc7-a31a-ca2080ae45d7-kube-api-access-xg8qb\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt\" (UID: \"10a62bca-19fb-4fc7-a31a-ca2080ae45d7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt"
Sep 17 18:31:10.725295 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:10.725258    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10a62bca-19fb-4fc7-a31a-ca2080ae45d7-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt\" (UID: \"10a62bca-19fb-4fc7-a31a-ca2080ae45d7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt"
Sep 17 18:31:10.725295 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:10.725297    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg8qb\" (UniqueName: \"kubernetes.io/projected/10a62bca-19fb-4fc7-a31a-ca2080ae45d7-kube-api-access-xg8qb\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt\" (UID: \"10a62bca-19fb-4fc7-a31a-ca2080ae45d7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt"
Sep 17 18:31:10.727910 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:10.727877    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10a62bca-19fb-4fc7-a31a-ca2080ae45d7-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt\" (UID: \"10a62bca-19fb-4fc7-a31a-ca2080ae45d7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt"
Sep 17 18:31:10.733195 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:10.733172    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg8qb\" (UniqueName: \"kubernetes.io/projected/10a62bca-19fb-4fc7-a31a-ca2080ae45d7-kube-api-access-xg8qb\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt\" (UID: \"10a62bca-19fb-4fc7-a31a-ca2080ae45d7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt"
Sep 17 18:31:10.813553 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:10.813451    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt"
Sep 17 18:31:11.088108 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:11.087998    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt"]
Sep 17 18:31:11.091584 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:31:11.091551    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a62bca_19fb_4fc7_a31a_ca2080ae45d7.slice/crio-5663e681c521cc23dc57ddcaf9121ec21c1a1e7f1cbe45e6ecb7371b4ee8e274 WatchSource:0}: Error finding container 5663e681c521cc23dc57ddcaf9121ec21c1a1e7f1cbe45e6ecb7371b4ee8e274: Status 404 returned error can't find the container with id 5663e681c521cc23dc57ddcaf9121ec21c1a1e7f1cbe45e6ecb7371b4ee8e274
Sep 17 18:31:11.105486 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:11.105447    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt" event={"ID":"10a62bca-19fb-4fc7-a31a-ca2080ae45d7","Type":"ContainerStarted","Data":"5663e681c521cc23dc57ddcaf9121ec21c1a1e7f1cbe45e6ecb7371b4ee8e274"}
Sep 17 18:31:12.112710 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:12.112676    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt" event={"ID":"10a62bca-19fb-4fc7-a31a-ca2080ae45d7","Type":"ContainerStarted","Data":"9818f5d5445e3ca1a960e966634b1110e0218887eea354bc34cb0eb91a5f89e7"}
Sep 17 18:31:12.113131 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:12.112803    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt"
Sep 17 18:31:12.129753 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:12.129692    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt" podStartSLOduration=2.12967658 podStartE2EDuration="2.12967658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:31:12.129145214 +0000 UTC m=+2060.219766867" watchObservedRunningTime="2026-09-17 18:31:12.12967658 +0000 UTC m=+2060.220298229"
Sep 17 18:31:14.400109 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:14.400036    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg" podUID="1e8ca83e-4fbc-45e5-998f-6df479737ce2" containerName="main" containerID="cri-o://c75f426f57222f5834609a6e87704ccbf5f3eea7aeee9734f05f1694bc511f49" gracePeriod=45
Sep 17 18:31:14.602697 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:14.602642    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg" podUID="1e8ca83e-4fbc-45e5-998f-6df479737ce2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 18:31:15.137220 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:15.137189    2581 generic.go:356] "Generic (PLEG): container finished" podID="1e8ca83e-4fbc-45e5-998f-6df479737ce2" containerID="c75f426f57222f5834609a6e87704ccbf5f3eea7aeee9734f05f1694bc511f49" exitCode=0
Sep 17 18:31:15.137341 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:15.137258    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg" event={"ID":"1e8ca83e-4fbc-45e5-998f-6df479737ce2","Type":"ContainerDied","Data":"c75f426f57222f5834609a6e87704ccbf5f3eea7aeee9734f05f1694bc511f49"}
Sep 17 18:31:15.155910 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:15.155877    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"
Sep 17 18:31:15.271122 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:15.271085    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e8ca83e-4fbc-45e5-998f-6df479737ce2-dshm\") pod \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\" (UID: \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\") "
Sep 17 18:31:15.271327 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:15.271157    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e8ca83e-4fbc-45e5-998f-6df479737ce2-tmp-dir\") pod \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\" (UID: \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\") "
Sep 17 18:31:15.271327 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:15.271194    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e8ca83e-4fbc-45e5-998f-6df479737ce2-model-cache\") pod \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\" (UID: \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\") "
Sep 17 18:31:15.271327 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:15.271214    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e8ca83e-4fbc-45e5-998f-6df479737ce2-cabundle-cert\") pod \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\" (UID: \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\") "
Sep 17 18:31:15.271327 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:15.271238    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e8ca83e-4fbc-45e5-998f-6df479737ce2-tls-certs\") pod \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\" (UID: \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\") "
Sep 17 18:31:15.271327 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:15.271257    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e8ca83e-4fbc-45e5-998f-6df479737ce2-kserve-provision-location\") pod \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\" (UID: \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\") "
Sep 17 18:31:15.271327 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:15.271292    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9h8v\" (UniqueName: \"kubernetes.io/projected/1e8ca83e-4fbc-45e5-998f-6df479737ce2-kube-api-access-b9h8v\") pod \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\" (UID: \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\") "
Sep 17 18:31:15.271655 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:15.271338    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e8ca83e-4fbc-45e5-998f-6df479737ce2-home\") pod \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\" (UID: \"1e8ca83e-4fbc-45e5-998f-6df479737ce2\") "
Sep 17 18:31:15.271655 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:15.271513    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e8ca83e-4fbc-45e5-998f-6df479737ce2-model-cache" (OuterVolumeSpecName: "model-cache") pod "1e8ca83e-4fbc-45e5-998f-6df479737ce2" (UID: "1e8ca83e-4fbc-45e5-998f-6df479737ce2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:31:15.271655 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:15.271638    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e8ca83e-4fbc-45e5-998f-6df479737ce2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1e8ca83e-4fbc-45e5-998f-6df479737ce2" (UID: "1e8ca83e-4fbc-45e5-998f-6df479737ce2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:31:15.271794 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:15.271714    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e8ca83e-4fbc-45e5-998f-6df479737ce2-model-cache\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:31:15.272142 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:15.272115    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e8ca83e-4fbc-45e5-998f-6df479737ce2-home" (OuterVolumeSpecName: "home") pod "1e8ca83e-4fbc-45e5-998f-6df479737ce2" (UID: "1e8ca83e-4fbc-45e5-998f-6df479737ce2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:31:15.273648 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:15.273619    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e8ca83e-4fbc-45e5-998f-6df479737ce2-kube-api-access-b9h8v" (OuterVolumeSpecName: "kube-api-access-b9h8v") pod "1e8ca83e-4fbc-45e5-998f-6df479737ce2" (UID: "1e8ca83e-4fbc-45e5-998f-6df479737ce2"). InnerVolumeSpecName "kube-api-access-b9h8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:31:15.273814 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:15.273788    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e8ca83e-4fbc-45e5-998f-6df479737ce2-dshm" (OuterVolumeSpecName: "dshm") pod "1e8ca83e-4fbc-45e5-998f-6df479737ce2" (UID: "1e8ca83e-4fbc-45e5-998f-6df479737ce2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:31:15.273937 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:15.273826    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e8ca83e-4fbc-45e5-998f-6df479737ce2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1e8ca83e-4fbc-45e5-998f-6df479737ce2" (UID: "1e8ca83e-4fbc-45e5-998f-6df479737ce2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:31:15.284207 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:15.284181    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e8ca83e-4fbc-45e5-998f-6df479737ce2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1e8ca83e-4fbc-45e5-998f-6df479737ce2" (UID: "1e8ca83e-4fbc-45e5-998f-6df479737ce2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:31:15.294079 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:15.294045    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e8ca83e-4fbc-45e5-998f-6df479737ce2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1e8ca83e-4fbc-45e5-998f-6df479737ce2" (UID: "1e8ca83e-4fbc-45e5-998f-6df479737ce2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:31:15.372561 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:15.372519    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e8ca83e-4fbc-45e5-998f-6df479737ce2-cabundle-cert\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:31:15.372561 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:15.372550    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e8ca83e-4fbc-45e5-998f-6df479737ce2-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:31:15.372561 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:15.372559    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e8ca83e-4fbc-45e5-998f-6df479737ce2-kserve-provision-location\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:31:15.372561 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:15.372568    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9h8v\" (UniqueName: \"kubernetes.io/projected/1e8ca83e-4fbc-45e5-998f-6df479737ce2-kube-api-access-b9h8v\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:31:15.372827 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:15.372578    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e8ca83e-4fbc-45e5-998f-6df479737ce2-home\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:31:15.372827 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:15.372587    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e8ca83e-4fbc-45e5-998f-6df479737ce2-dshm\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:31:15.372827 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:15.372596    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e8ca83e-4fbc-45e5-998f-6df479737ce2-tmp-dir\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:31:16.144256 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:16.144211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg" event={"ID":"1e8ca83e-4fbc-45e5-998f-6df479737ce2","Type":"ContainerDied","Data":"c9cb62b7a6eeed7e5a036b2fd4be271a156616087272343dec771cb20bfa4ea4"}
Sep 17 18:31:16.144256 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:16.144256    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"
Sep 17 18:31:16.144762 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:16.144285    2581 scope.go:117] "RemoveContainer" containerID="c75f426f57222f5834609a6e87704ccbf5f3eea7aeee9734f05f1694bc511f49"
Sep 17 18:31:16.153438 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:16.153418    2581 scope.go:117] "RemoveContainer" containerID="fd34503f6764a84e8f09849e3389ce89648cff1747bb70be8902d5a5f69232fe"
Sep 17 18:31:16.169115 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:16.169089    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"]
Sep 17 18:31:16.174510 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:16.174488    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-868cfb54f76gkrg"]
Sep 17 18:31:16.358235 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:16.358192    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e8ca83e-4fbc-45e5-998f-6df479737ce2" path="/var/lib/kubelet/pods/1e8ca83e-4fbc-45e5-998f-6df479737ce2/volumes"
Sep 17 18:31:43.119013 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:43.118977    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt"
Sep 17 18:31:44.678243 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:44.678208    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt"]
Sep 17 18:31:52.495369 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:52.495330    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/2.log"
Sep 17 18:31:52.507702 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:52.507677    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/2.log"
Sep 17 18:31:53.119153 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:53.119112    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt" podUID="10a62bca-19fb-4fc7-a31a-ca2080ae45d7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:31:59.698468 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:31:59.698411    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt" podUID="10a62bca-19fb-4fc7-a31a-ca2080ae45d7" containerName="main" containerID="cri-o://9818f5d5445e3ca1a960e966634b1110e0218887eea354bc34cb0eb91a5f89e7" gracePeriod=45
Sep 17 18:32:03.120004 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:03.119962    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt" podUID="10a62bca-19fb-4fc7-a31a-ca2080ae45d7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:32:09.312567 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:09.312526    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p"]
Sep 17 18:32:09.313177 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:09.313154    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e8ca83e-4fbc-45e5-998f-6df479737ce2" containerName="storage-initializer"
Sep 17 18:32:09.313225 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:09.313182    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e8ca83e-4fbc-45e5-998f-6df479737ce2" containerName="storage-initializer"
Sep 17 18:32:09.313225 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:09.313202    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e8ca83e-4fbc-45e5-998f-6df479737ce2" containerName="main"
Sep 17 18:32:09.313225 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:09.313211    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e8ca83e-4fbc-45e5-998f-6df479737ce2" containerName="main"
Sep 17 18:32:09.313349 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:09.313335    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e8ca83e-4fbc-45e5-998f-6df479737ce2" containerName="main"
Sep 17 18:32:09.315743 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:09.315723    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p"
Sep 17 18:32:09.319618 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:09.319589    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 17 18:32:09.319774 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:09.319645    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-fr6c6\""
Sep 17 18:32:09.319774 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:09.319663    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 17 18:32:09.319774 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:09.319647    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisvc-model-fb-opt-125m-oci-r-0fba19de-epp-sa-dockercfg-n8q2k\""
Sep 17 18:32:09.319774 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:09.319753    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 17 18:32:09.326503 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:09.326133    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p"]
Sep 17 18:32:09.449559 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:09.449509    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr9r6\" (UniqueName: \"kubernetes.io/projected/3b4f49da-e1aa-495e-a32e-9e65e9624c22-kube-api-access-cr9r6\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p\" (UID: \"3b4f49da-e1aa-495e-a32e-9e65e9624c22\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p"
Sep 17 18:32:09.449750 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:09.449575    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b4f49da-e1aa-495e-a32e-9e65e9624c22-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p\" (UID: \"3b4f49da-e1aa-495e-a32e-9e65e9624c22\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p"
Sep 17 18:32:09.550759 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:09.550715    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr9r6\" (UniqueName: \"kubernetes.io/projected/3b4f49da-e1aa-495e-a32e-9e65e9624c22-kube-api-access-cr9r6\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p\" (UID: \"3b4f49da-e1aa-495e-a32e-9e65e9624c22\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p"
Sep 17 18:32:09.550759 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:09.550765    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b4f49da-e1aa-495e-a32e-9e65e9624c22-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p\" (UID: \"3b4f49da-e1aa-495e-a32e-9e65e9624c22\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p"
Sep 17 18:32:09.553384 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:09.553355    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b4f49da-e1aa-495e-a32e-9e65e9624c22-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p\" (UID: \"3b4f49da-e1aa-495e-a32e-9e65e9624c22\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p"
Sep 17 18:32:09.559738 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:09.559707    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr9r6\" (UniqueName: \"kubernetes.io/projected/3b4f49da-e1aa-495e-a32e-9e65e9624c22-kube-api-access-cr9r6\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p\" (UID: \"3b4f49da-e1aa-495e-a32e-9e65e9624c22\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p"
Sep 17 18:32:09.630527 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:09.630434    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p"
Sep 17 18:32:09.904304 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:09.904269    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p"]
Sep 17 18:32:09.906845 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:32:09.906809    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b4f49da_e1aa_495e_a32e_9e65e9624c22.slice/crio-51e5d2b0375d8bbeaa4232ced6e7af91c1e8b54895c2d804f08264a27d7a0b60 WatchSource:0}: Error finding container 51e5d2b0375d8bbeaa4232ced6e7af91c1e8b54895c2d804f08264a27d7a0b60: Status 404 returned error can't find the container with id 51e5d2b0375d8bbeaa4232ced6e7af91c1e8b54895c2d804f08264a27d7a0b60
Sep 17 18:32:10.402476 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:10.402440    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p" event={"ID":"3b4f49da-e1aa-495e-a32e-9e65e9624c22","Type":"ContainerStarted","Data":"1b6049a2504251150e656f89884c77cba1239f6afbb6fc9a7e4d35cff0b362d6"}
Sep 17 18:32:10.402886 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:10.402602    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p"
Sep 17 18:32:10.402886 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:10.402638    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p" event={"ID":"3b4f49da-e1aa-495e-a32e-9e65e9624c22","Type":"ContainerStarted","Data":"51e5d2b0375d8bbeaa4232ced6e7af91c1e8b54895c2d804f08264a27d7a0b60"}
Sep 17 18:32:10.421807 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:10.421735    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p" podStartSLOduration=1.421714046 podStartE2EDuration="1.421714046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:32:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:32:10.418986444 +0000 UTC m=+2118.509608095" watchObservedRunningTime="2026-09-17 18:32:10.421714046 +0000 UTC m=+2118.512335699"
Sep 17 18:32:13.119340 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:13.119294    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt" podUID="10a62bca-19fb-4fc7-a31a-ca2080ae45d7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:32:13.119713 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:13.119400    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt"
Sep 17 18:32:23.119786 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:23.119740    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt" podUID="10a62bca-19fb-4fc7-a31a-ca2080ae45d7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:32:29.956646 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:29.956618    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt"
Sep 17 18:32:30.036863 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:30.036826    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xg8qb\" (UniqueName: \"kubernetes.io/projected/10a62bca-19fb-4fc7-a31a-ca2080ae45d7-kube-api-access-xg8qb\") pod \"10a62bca-19fb-4fc7-a31a-ca2080ae45d7\" (UID: \"10a62bca-19fb-4fc7-a31a-ca2080ae45d7\") "
Sep 17 18:32:30.037062 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:30.037014    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10a62bca-19fb-4fc7-a31a-ca2080ae45d7-tls-certs\") pod \"10a62bca-19fb-4fc7-a31a-ca2080ae45d7\" (UID: \"10a62bca-19fb-4fc7-a31a-ca2080ae45d7\") "
Sep 17 18:32:30.039223 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:30.039188    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10a62bca-19fb-4fc7-a31a-ca2080ae45d7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "10a62bca-19fb-4fc7-a31a-ca2080ae45d7" (UID: "10a62bca-19fb-4fc7-a31a-ca2080ae45d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:32:30.039223 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:30.039205    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a62bca-19fb-4fc7-a31a-ca2080ae45d7-kube-api-access-xg8qb" (OuterVolumeSpecName: "kube-api-access-xg8qb") pod "10a62bca-19fb-4fc7-a31a-ca2080ae45d7" (UID: "10a62bca-19fb-4fc7-a31a-ca2080ae45d7"). InnerVolumeSpecName "kube-api-access-xg8qb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:32:30.138261 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:30.138168    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10a62bca-19fb-4fc7-a31a-ca2080ae45d7-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:32:30.138261 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:30.138202    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg8qb\" (UniqueName: \"kubernetes.io/projected/10a62bca-19fb-4fc7-a31a-ca2080ae45d7-kube-api-access-xg8qb\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:32:30.496968 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:30.496933    2581 generic.go:356] "Generic (PLEG): container finished" podID="10a62bca-19fb-4fc7-a31a-ca2080ae45d7" containerID="9818f5d5445e3ca1a960e966634b1110e0218887eea354bc34cb0eb91a5f89e7" exitCode=0
Sep 17 18:32:30.497149 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:30.496981    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt"
Sep 17 18:32:30.497149 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:30.497019    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt" event={"ID":"10a62bca-19fb-4fc7-a31a-ca2080ae45d7","Type":"ContainerDied","Data":"9818f5d5445e3ca1a960e966634b1110e0218887eea354bc34cb0eb91a5f89e7"}
Sep 17 18:32:30.497149 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:30.497078    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt" event={"ID":"10a62bca-19fb-4fc7-a31a-ca2080ae45d7","Type":"ContainerDied","Data":"5663e681c521cc23dc57ddcaf9121ec21c1a1e7f1cbe45e6ecb7371b4ee8e274"}
Sep 17 18:32:30.497149 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:30.497106    2581 scope.go:117] "RemoveContainer" containerID="9818f5d5445e3ca1a960e966634b1110e0218887eea354bc34cb0eb91a5f89e7"
Sep 17 18:32:30.505699 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:30.505680    2581 scope.go:117] "RemoveContainer" containerID="9818f5d5445e3ca1a960e966634b1110e0218887eea354bc34cb0eb91a5f89e7"
Sep 17 18:32:30.505991 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:32:30.505971    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9818f5d5445e3ca1a960e966634b1110e0218887eea354bc34cb0eb91a5f89e7\": container with ID starting with 9818f5d5445e3ca1a960e966634b1110e0218887eea354bc34cb0eb91a5f89e7 not found: ID does not exist" containerID="9818f5d5445e3ca1a960e966634b1110e0218887eea354bc34cb0eb91a5f89e7"
Sep 17 18:32:30.506072 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:30.505998    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9818f5d5445e3ca1a960e966634b1110e0218887eea354bc34cb0eb91a5f89e7"} err="failed to get container status \"9818f5d5445e3ca1a960e966634b1110e0218887eea354bc34cb0eb91a5f89e7\": rpc error: code = NotFound desc = could not find container \"9818f5d5445e3ca1a960e966634b1110e0218887eea354bc34cb0eb91a5f89e7\": container with ID starting with 9818f5d5445e3ca1a960e966634b1110e0218887eea354bc34cb0eb91a5f89e7 not found: ID does not exist"
Sep 17 18:32:30.515208 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:30.515180    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt"]
Sep 17 18:32:30.518005 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:30.517982    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheh5pnt"]
Sep 17 18:32:32.358090 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:32.358058    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10a62bca-19fb-4fc7-a31a-ca2080ae45d7" path="/var/lib/kubelet/pods/10a62bca-19fb-4fc7-a31a-ca2080ae45d7/volumes"
Sep 17 18:32:40.936518 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:40.936482    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm"]
Sep 17 18:32:41.408480 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:41.408447    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p"
Sep 17 18:32:47.902348 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:47.902260    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm" podUID="683cd26d-2b55-404d-b799-1032049756cb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:32:55.955515 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:55.955472    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm" podUID="683cd26d-2b55-404d-b799-1032049756cb" containerName="main" containerID="cri-o://5d96e644434a9505b459dce57cac6d8adf18668295b654d8bda432ca31c34654" gracePeriod=45
Sep 17 18:32:57.902064 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:32:57.902025    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm" podUID="683cd26d-2b55-404d-b799-1032049756cb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:33:07.902238 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:07.902197    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm" podUID="683cd26d-2b55-404d-b799-1032049756cb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:33:07.902640 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:07.902339    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm"
Sep 17 18:33:17.902615 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:17.902574    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm" podUID="683cd26d-2b55-404d-b799-1032049756cb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:33:26.203290 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:26.203259    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm"
Sep 17 18:33:26.259537 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:26.259451    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/683cd26d-2b55-404d-b799-1032049756cb-tls-certs\") pod \"683cd26d-2b55-404d-b799-1032049756cb\" (UID: \"683cd26d-2b55-404d-b799-1032049756cb\") "
Sep 17 18:33:26.259537 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:26.259508    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dscjw\" (UniqueName: \"kubernetes.io/projected/683cd26d-2b55-404d-b799-1032049756cb-kube-api-access-dscjw\") pod \"683cd26d-2b55-404d-b799-1032049756cb\" (UID: \"683cd26d-2b55-404d-b799-1032049756cb\") "
Sep 17 18:33:26.261649 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:26.261606    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/683cd26d-2b55-404d-b799-1032049756cb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "683cd26d-2b55-404d-b799-1032049756cb" (UID: "683cd26d-2b55-404d-b799-1032049756cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:33:26.261767 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:26.261706    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/683cd26d-2b55-404d-b799-1032049756cb-kube-api-access-dscjw" (OuterVolumeSpecName: "kube-api-access-dscjw") pod "683cd26d-2b55-404d-b799-1032049756cb" (UID: "683cd26d-2b55-404d-b799-1032049756cb"). InnerVolumeSpecName "kube-api-access-dscjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:33:26.360784 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:26.360751    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/683cd26d-2b55-404d-b799-1032049756cb-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:33:26.360784 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:26.360777    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dscjw\" (UniqueName: \"kubernetes.io/projected/683cd26d-2b55-404d-b799-1032049756cb-kube-api-access-dscjw\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:33:26.760046 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:26.760015    2581 generic.go:356] "Generic (PLEG): container finished" podID="683cd26d-2b55-404d-b799-1032049756cb" containerID="5d96e644434a9505b459dce57cac6d8adf18668295b654d8bda432ca31c34654" exitCode=0
Sep 17 18:33:26.760253 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:26.760061    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm"
Sep 17 18:33:26.760253 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:26.760104    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm" event={"ID":"683cd26d-2b55-404d-b799-1032049756cb","Type":"ContainerDied","Data":"5d96e644434a9505b459dce57cac6d8adf18668295b654d8bda432ca31c34654"}
Sep 17 18:33:26.760253 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:26.760165    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm" event={"ID":"683cd26d-2b55-404d-b799-1032049756cb","Type":"ContainerDied","Data":"48f52f03f0881c5c0fe5c1557f6ac8fb41436189bdc5e9a37c0d96241ba1404f"}
Sep 17 18:33:26.760253 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:26.760200    2581 scope.go:117] "RemoveContainer" containerID="5d96e644434a9505b459dce57cac6d8adf18668295b654d8bda432ca31c34654"
Sep 17 18:33:26.769315 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:26.769276    2581 scope.go:117] "RemoveContainer" containerID="5d96e644434a9505b459dce57cac6d8adf18668295b654d8bda432ca31c34654"
Sep 17 18:33:26.769623 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:33:26.769597    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d96e644434a9505b459dce57cac6d8adf18668295b654d8bda432ca31c34654\": container with ID starting with 5d96e644434a9505b459dce57cac6d8adf18668295b654d8bda432ca31c34654 not found: ID does not exist" containerID="5d96e644434a9505b459dce57cac6d8adf18668295b654d8bda432ca31c34654"
Sep 17 18:33:26.769678 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:26.769632    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d96e644434a9505b459dce57cac6d8adf18668295b654d8bda432ca31c34654"} err="failed to get container status \"5d96e644434a9505b459dce57cac6d8adf18668295b654d8bda432ca31c34654\": rpc error: code = NotFound desc = could not find container \"5d96e644434a9505b459dce57cac6d8adf18668295b654d8bda432ca31c34654\": container with ID starting with 5d96e644434a9505b459dce57cac6d8adf18668295b654d8bda432ca31c34654 not found: ID does not exist"
Sep 17 18:33:26.777036 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:26.777013    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm"]
Sep 17 18:33:26.780547 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:26.780526    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-scheh48xm"]
Sep 17 18:33:28.358502 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:28.358469    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="683cd26d-2b55-404d-b799-1032049756cb" path="/var/lib/kubelet/pods/683cd26d-2b55-404d-b799-1032049756cb/volumes"
Sep 17 18:33:44.578738 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.578696    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"]
Sep 17 18:33:44.579184 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.579117    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10a62bca-19fb-4fc7-a31a-ca2080ae45d7" containerName="main"
Sep 17 18:33:44.579184 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.579133    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="10a62bca-19fb-4fc7-a31a-ca2080ae45d7" containerName="main"
Sep 17 18:33:44.579184 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.579144    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="683cd26d-2b55-404d-b799-1032049756cb" containerName="main"
Sep 17 18:33:44.579184 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.579149    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="683cd26d-2b55-404d-b799-1032049756cb" containerName="main"
Sep 17 18:33:44.579319 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.579232    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="683cd26d-2b55-404d-b799-1032049756cb" containerName="main"
Sep 17 18:33:44.579319 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.579240    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="10a62bca-19fb-4fc7-a31a-ca2080ae45d7" containerName="main"
Sep 17 18:33:44.582316 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.582295    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"
Sep 17 18:33:44.584741 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.584712    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-5fnxb\""
Sep 17 18:33:44.584936 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.584885    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 17 18:33:44.585053 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.584892    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Sep 17 18:33:44.585213 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.585191    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:33:44.585213 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.585207    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 17 18:33:44.585390 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.585270    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 17 18:33:44.593874 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.593850    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"]
Sep 17 18:33:44.721791 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.721757    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffedf828-ea72-412c-bd99-6c1bacd01bf2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw\" (UID: \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"
Sep 17 18:33:44.721791 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.721795    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffedf828-ea72-412c-bd99-6c1bacd01bf2-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw\" (UID: \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"
Sep 17 18:33:44.722041 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.721817    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffedf828-ea72-412c-bd99-6c1bacd01bf2-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw\" (UID: \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"
Sep 17 18:33:44.722041 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.721836    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtp4w\" (UniqueName: \"kubernetes.io/projected/ffedf828-ea72-412c-bd99-6c1bacd01bf2-kube-api-access-wtp4w\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw\" (UID: \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"
Sep 17 18:33:44.722041 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.721936    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffedf828-ea72-412c-bd99-6c1bacd01bf2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw\" (UID: \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"
Sep 17 18:33:44.722041 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.721972    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffedf828-ea72-412c-bd99-6c1bacd01bf2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw\" (UID: \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"
Sep 17 18:33:44.722041 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.722024    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffedf828-ea72-412c-bd99-6c1bacd01bf2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw\" (UID: \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"
Sep 17 18:33:44.722202 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.722054    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffedf828-ea72-412c-bd99-6c1bacd01bf2-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw\" (UID: \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"
Sep 17 18:33:44.777520 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.777477    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl"]
Sep 17 18:33:44.780973 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.780951    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl"
Sep 17 18:33:44.783469 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.783438    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-rggv8\""
Sep 17 18:33:44.789147 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.789121    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl"]
Sep 17 18:33:44.822674 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.822635    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffedf828-ea72-412c-bd99-6c1bacd01bf2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw\" (UID: \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"
Sep 17 18:33:44.822674 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.822689    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffedf828-ea72-412c-bd99-6c1bacd01bf2-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw\" (UID: \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"
Sep 17 18:33:44.822961 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.822747    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffedf828-ea72-412c-bd99-6c1bacd01bf2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw\" (UID: \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"
Sep 17 18:33:44.822961 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.822766    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffedf828-ea72-412c-bd99-6c1bacd01bf2-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw\" (UID: \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"
Sep 17 18:33:44.822961 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.822787    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffedf828-ea72-412c-bd99-6c1bacd01bf2-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw\" (UID: \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"
Sep 17 18:33:44.822961 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.822806    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtp4w\" (UniqueName: \"kubernetes.io/projected/ffedf828-ea72-412c-bd99-6c1bacd01bf2-kube-api-access-wtp4w\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw\" (UID: \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"
Sep 17 18:33:44.822961 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.822838    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffedf828-ea72-412c-bd99-6c1bacd01bf2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw\" (UID: \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"
Sep 17 18:33:44.822961 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.822865    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffedf828-ea72-412c-bd99-6c1bacd01bf2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw\" (UID: \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"
Sep 17 18:33:44.823325 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.823298    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffedf828-ea72-412c-bd99-6c1bacd01bf2-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw\" (UID: \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"
Sep 17 18:33:44.823445 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.823365    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffedf828-ea72-412c-bd99-6c1bacd01bf2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw\" (UID: \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"
Sep 17 18:33:44.823445 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.823384    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffedf828-ea72-412c-bd99-6c1bacd01bf2-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw\" (UID: \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"
Sep 17 18:33:44.823445 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.823434    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffedf828-ea72-412c-bd99-6c1bacd01bf2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw\" (UID: \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"
Sep 17 18:33:44.823628 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.823491    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffedf828-ea72-412c-bd99-6c1bacd01bf2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw\" (UID: \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"
Sep 17 18:33:44.825271 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.825249    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffedf828-ea72-412c-bd99-6c1bacd01bf2-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw\" (UID: \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"
Sep 17 18:33:44.825639 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.825616    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffedf828-ea72-412c-bd99-6c1bacd01bf2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw\" (UID: \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"
Sep 17 18:33:44.830490 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.830429    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtp4w\" (UniqueName: \"kubernetes.io/projected/ffedf828-ea72-412c-bd99-6c1bacd01bf2-kube-api-access-wtp4w\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw\" (UID: \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"
Sep 17 18:33:44.895250 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.895205    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"
Sep 17 18:33:44.923544 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.923508    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cxj5\" (UniqueName: \"kubernetes.io/projected/a8e1b2af-0b5a-44f4-a204-3c45e6ee64be-kube-api-access-8cxj5\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl\" (UID: \"a8e1b2af-0b5a-44f4-a204-3c45e6ee64be\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl"
Sep 17 18:33:44.923740 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:44.923553    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8e1b2af-0b5a-44f4-a204-3c45e6ee64be-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl\" (UID: \"a8e1b2af-0b5a-44f4-a204-3c45e6ee64be\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl"
Sep 17 18:33:45.025064 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:45.025022    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cxj5\" (UniqueName: \"kubernetes.io/projected/a8e1b2af-0b5a-44f4-a204-3c45e6ee64be-kube-api-access-8cxj5\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl\" (UID: \"a8e1b2af-0b5a-44f4-a204-3c45e6ee64be\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl"
Sep 17 18:33:45.025064 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:45.025068    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8e1b2af-0b5a-44f4-a204-3c45e6ee64be-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl\" (UID: \"a8e1b2af-0b5a-44f4-a204-3c45e6ee64be\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl"
Sep 17 18:33:45.027619 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:45.027592    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8e1b2af-0b5a-44f4-a204-3c45e6ee64be-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl\" (UID: \"a8e1b2af-0b5a-44f4-a204-3c45e6ee64be\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl"
Sep 17 18:33:45.034007 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:45.033972    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cxj5\" (UniqueName: \"kubernetes.io/projected/a8e1b2af-0b5a-44f4-a204-3c45e6ee64be-kube-api-access-8cxj5\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl\" (UID: \"a8e1b2af-0b5a-44f4-a204-3c45e6ee64be\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl"
Sep 17 18:33:45.092025 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:45.091926    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl"
Sep 17 18:33:45.196781 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:45.196753    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"]
Sep 17 18:33:45.198935 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:33:45.198882    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffedf828_ea72_412c_bd99_6c1bacd01bf2.slice/crio-bf53efdcddbaee3e65cb595c5348ca85c437324b3c01fa6e866cd9c7c08f98f7 WatchSource:0}: Error finding container bf53efdcddbaee3e65cb595c5348ca85c437324b3c01fa6e866cd9c7c08f98f7: Status 404 returned error can't find the container with id bf53efdcddbaee3e65cb595c5348ca85c437324b3c01fa6e866cd9c7c08f98f7
Sep 17 18:33:45.364511 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:45.364481    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl"]
Sep 17 18:33:45.366682 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:33:45.366649    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e1b2af_0b5a_44f4_a204_3c45e6ee64be.slice/crio-8067833c135b5e86186ee6f2a27fbbd7abd80266b705c49a4b77cfc030a53159 WatchSource:0}: Error finding container 8067833c135b5e86186ee6f2a27fbbd7abd80266b705c49a4b77cfc030a53159: Status 404 returned error can't find the container with id 8067833c135b5e86186ee6f2a27fbbd7abd80266b705c49a4b77cfc030a53159
Sep 17 18:33:45.854875 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:45.854832    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl" event={"ID":"a8e1b2af-0b5a-44f4-a204-3c45e6ee64be","Type":"ContainerStarted","Data":"0f2438b2fd29abc07784db2a575ed9f15e644b5c4d3555c0e8bf4c4a3284c7a2"}
Sep 17 18:33:45.855381 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:45.854937    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl" event={"ID":"a8e1b2af-0b5a-44f4-a204-3c45e6ee64be","Type":"ContainerStarted","Data":"8067833c135b5e86186ee6f2a27fbbd7abd80266b705c49a4b77cfc030a53159"}
Sep 17 18:33:45.855381 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:45.855194    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl"
Sep 17 18:33:45.857467 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:45.857433    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw" event={"ID":"ffedf828-ea72-412c-bd99-6c1bacd01bf2","Type":"ContainerStarted","Data":"f2aac0aa8750c3d0d10d15c18cb24c9d26bbe851700216062f389e7a4aa3176a"}
Sep 17 18:33:45.857621 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:45.857490    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw" event={"ID":"ffedf828-ea72-412c-bd99-6c1bacd01bf2","Type":"ContainerStarted","Data":"bf53efdcddbaee3e65cb595c5348ca85c437324b3c01fa6e866cd9c7c08f98f7"}
Sep 17 18:33:45.878870 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:45.878779    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl" podStartSLOduration=1.878755473 podStartE2EDuration="1.878755473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:33:45.876330593 +0000 UTC m=+2213.966952246" watchObservedRunningTime="2026-09-17 18:33:45.878755473 +0000 UTC m=+2213.969377123"
Sep 17 18:33:48.873702 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:48.873670    2581 generic.go:356] "Generic (PLEG): container finished" podID="ffedf828-ea72-412c-bd99-6c1bacd01bf2" containerID="f2aac0aa8750c3d0d10d15c18cb24c9d26bbe851700216062f389e7a4aa3176a" exitCode=0
Sep 17 18:33:48.874156 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:48.873743    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw" event={"ID":"ffedf828-ea72-412c-bd99-6c1bacd01bf2","Type":"ContainerDied","Data":"f2aac0aa8750c3d0d10d15c18cb24c9d26bbe851700216062f389e7a4aa3176a"}
Sep 17 18:33:49.880077 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:49.880039    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw" event={"ID":"ffedf828-ea72-412c-bd99-6c1bacd01bf2","Type":"ContainerStarted","Data":"6665327c20c794d71bfb12f9b266d3a659c6e1950784f69a3b34939244ef28ec"}
Sep 17 18:33:49.895344 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:49.895318    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"
Sep 17 18:33:49.926695 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:49.926636    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw" podStartSLOduration=3.059331124 podStartE2EDuration="5.92662158s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.867290456s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:33:49.923711552 +0000 UTC m=+2218.014333206" watchObservedRunningTime="2026-09-17 18:33:49.92662158 +0000 UTC m=+2218.017243229"
Sep 17 18:33:54.896156 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:54.896120    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"
Sep 17 18:33:54.897780 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:33:54.897750    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw" podUID="ffedf828-ea72-412c-bd99-6c1bacd01bf2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Sep 17 18:34:04.896557 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:34:04.896501    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw" podUID="ffedf828-ea72-412c-bd99-6c1bacd01bf2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Sep 17 18:34:14.895758 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:34:14.895716    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw" podUID="ffedf828-ea72-412c-bd99-6c1bacd01bf2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Sep 17 18:34:16.864557 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:34:16.864519    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl"
Sep 17 18:34:24.429457 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:34:24.429419    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p"]
Sep 17 18:34:24.895915 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:34:24.895858    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw" podUID="ffedf828-ea72-412c-bd99-6c1bacd01bf2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Sep 17 18:34:31.409305 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:34:31.409261    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p" podUID="3b4f49da-e1aa-495e-a32e-9e65e9624c22" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:34:34.895882 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:34:34.895827    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw" podUID="ffedf828-ea72-412c-bd99-6c1bacd01bf2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Sep 17 18:34:39.450824 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:34:39.450781    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p" podUID="3b4f49da-e1aa-495e-a32e-9e65e9624c22" containerName="main" containerID="cri-o://1b6049a2504251150e656f89884c77cba1239f6afbb6fc9a7e4d35cff0b362d6" gracePeriod=45
Sep 17 18:34:41.408390 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:34:41.408345    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p" podUID="3b4f49da-e1aa-495e-a32e-9e65e9624c22" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:34:44.896117 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:34:44.896070    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw" podUID="ffedf828-ea72-412c-bd99-6c1bacd01bf2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Sep 17 18:34:51.408508 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:34:51.408454    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p" podUID="3b4f49da-e1aa-495e-a32e-9e65e9624c22" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:34:51.408893 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:34:51.408590    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p"
Sep 17 18:34:54.895857 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:34:54.895806    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw" podUID="ffedf828-ea72-412c-bd99-6c1bacd01bf2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Sep 17 18:35:01.408626 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:01.408586    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p" podUID="3b4f49da-e1aa-495e-a32e-9e65e9624c22" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:35:04.905799 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:04.905758    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"
Sep 17 18:35:04.914023 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:04.913992    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"
Sep 17 18:35:09.714708 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:09.714626    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p"
Sep 17 18:35:09.826317 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:09.826281    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cr9r6\" (UniqueName: \"kubernetes.io/projected/3b4f49da-e1aa-495e-a32e-9e65e9624c22-kube-api-access-cr9r6\") pod \"3b4f49da-e1aa-495e-a32e-9e65e9624c22\" (UID: \"3b4f49da-e1aa-495e-a32e-9e65e9624c22\") "
Sep 17 18:35:09.826529 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:09.826342    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b4f49da-e1aa-495e-a32e-9e65e9624c22-tls-certs\") pod \"3b4f49da-e1aa-495e-a32e-9e65e9624c22\" (UID: \"3b4f49da-e1aa-495e-a32e-9e65e9624c22\") "
Sep 17 18:35:09.828648 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:09.828605    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b4f49da-e1aa-495e-a32e-9e65e9624c22-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3b4f49da-e1aa-495e-a32e-9e65e9624c22" (UID: "3b4f49da-e1aa-495e-a32e-9e65e9624c22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:35:09.828812 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:09.828675    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b4f49da-e1aa-495e-a32e-9e65e9624c22-kube-api-access-cr9r6" (OuterVolumeSpecName: "kube-api-access-cr9r6") pod "3b4f49da-e1aa-495e-a32e-9e65e9624c22" (UID: "3b4f49da-e1aa-495e-a32e-9e65e9624c22"). InnerVolumeSpecName "kube-api-access-cr9r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:35:09.927257 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:09.927217    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr9r6\" (UniqueName: \"kubernetes.io/projected/3b4f49da-e1aa-495e-a32e-9e65e9624c22-kube-api-access-cr9r6\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:35:09.927257 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:09.927250    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b4f49da-e1aa-495e-a32e-9e65e9624c22-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:35:10.295020 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:10.294978    2581 generic.go:356] "Generic (PLEG): container finished" podID="3b4f49da-e1aa-495e-a32e-9e65e9624c22" containerID="1b6049a2504251150e656f89884c77cba1239f6afbb6fc9a7e4d35cff0b362d6" exitCode=0
Sep 17 18:35:10.295020 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:10.295023    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p"
Sep 17 18:35:10.295293 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:10.295063    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p" event={"ID":"3b4f49da-e1aa-495e-a32e-9e65e9624c22","Type":"ContainerDied","Data":"1b6049a2504251150e656f89884c77cba1239f6afbb6fc9a7e4d35cff0b362d6"}
Sep 17 18:35:10.295293 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:10.295124    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p" event={"ID":"3b4f49da-e1aa-495e-a32e-9e65e9624c22","Type":"ContainerDied","Data":"51e5d2b0375d8bbeaa4232ced6e7af91c1e8b54895c2d804f08264a27d7a0b60"}
Sep 17 18:35:10.295293 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:10.295154    2581 scope.go:117] "RemoveContainer" containerID="1b6049a2504251150e656f89884c77cba1239f6afbb6fc9a7e4d35cff0b362d6"
Sep 17 18:35:10.305002 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:10.304980    2581 scope.go:117] "RemoveContainer" containerID="1b6049a2504251150e656f89884c77cba1239f6afbb6fc9a7e4d35cff0b362d6"
Sep 17 18:35:10.305350 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:35:10.305324    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b6049a2504251150e656f89884c77cba1239f6afbb6fc9a7e4d35cff0b362d6\": container with ID starting with 1b6049a2504251150e656f89884c77cba1239f6afbb6fc9a7e4d35cff0b362d6 not found: ID does not exist" containerID="1b6049a2504251150e656f89884c77cba1239f6afbb6fc9a7e4d35cff0b362d6"
Sep 17 18:35:10.305467 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:10.305355    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b6049a2504251150e656f89884c77cba1239f6afbb6fc9a7e4d35cff0b362d6"} err="failed to get container status \"1b6049a2504251150e656f89884c77cba1239f6afbb6fc9a7e4d35cff0b362d6\": rpc error: code = NotFound desc = could not find container \"1b6049a2504251150e656f89884c77cba1239f6afbb6fc9a7e4d35cff0b362d6\": container with ID starting with 1b6049a2504251150e656f89884c77cba1239f6afbb6fc9a7e4d35cff0b362d6 not found: ID does not exist"
Sep 17 18:35:10.319248 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:10.319212    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p"]
Sep 17 18:35:10.321891 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:10.321862    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemzv5p"]
Sep 17 18:35:10.359204 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:10.359171    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b4f49da-e1aa-495e-a32e-9e65e9624c22" path="/var/lib/kubelet/pods/3b4f49da-e1aa-495e-a32e-9e65e9624c22/volumes"
Sep 17 18:35:44.318783 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:44.318739    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg"]
Sep 17 18:35:44.319300 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:44.319143    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b4f49da-e1aa-495e-a32e-9e65e9624c22" containerName="main"
Sep 17 18:35:44.319300 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:44.319155    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b4f49da-e1aa-495e-a32e-9e65e9624c22" containerName="main"
Sep 17 18:35:44.319300 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:44.319216    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b4f49da-e1aa-495e-a32e-9e65e9624c22" containerName="main"
Sep 17 18:35:44.321204 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:44.321181    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg"
Sep 17 18:35:44.324046 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:44.324021    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 17 18:35:44.324198 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:44.324029    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-lztdr\""
Sep 17 18:35:44.325186 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:44.325161    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 17 18:35:44.325339 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:44.325187    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 17 18:35:44.325339 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:44.325239    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-p6c9l\""
Sep 17 18:35:44.330657 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:44.330605    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg"]
Sep 17 18:35:44.445216 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:44.445163    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt49l\" (UniqueName: \"kubernetes.io/projected/a53f199e-e9ab-4d39-93b9-704fe1c6658a-kube-api-access-xt49l\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg\" (UID: \"a53f199e-e9ab-4d39-93b9-704fe1c6658a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg"
Sep 17 18:35:44.445216 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:44.445216    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a53f199e-e9ab-4d39-93b9-704fe1c6658a-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg\" (UID: \"a53f199e-e9ab-4d39-93b9-704fe1c6658a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg"
Sep 17 18:35:44.546288 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:44.546247    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt49l\" (UniqueName: \"kubernetes.io/projected/a53f199e-e9ab-4d39-93b9-704fe1c6658a-kube-api-access-xt49l\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg\" (UID: \"a53f199e-e9ab-4d39-93b9-704fe1c6658a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg"
Sep 17 18:35:44.546288 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:44.546290    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a53f199e-e9ab-4d39-93b9-704fe1c6658a-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg\" (UID: \"a53f199e-e9ab-4d39-93b9-704fe1c6658a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg"
Sep 17 18:35:44.548788 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:44.548763    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a53f199e-e9ab-4d39-93b9-704fe1c6658a-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg\" (UID: \"a53f199e-e9ab-4d39-93b9-704fe1c6658a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg"
Sep 17 18:35:44.553770 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:44.553747    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt49l\" (UniqueName: \"kubernetes.io/projected/a53f199e-e9ab-4d39-93b9-704fe1c6658a-kube-api-access-xt49l\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg\" (UID: \"a53f199e-e9ab-4d39-93b9-704fe1c6658a\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg"
Sep 17 18:35:44.634602 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:44.634518    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg"
Sep 17 18:35:44.905676 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:44.905646    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg"]
Sep 17 18:35:44.907420 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:35:44.907392    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53f199e_e9ab_4d39_93b9_704fe1c6658a.slice/crio-609b45ad1520b8fd10f0be64867da3b734052a1df82c76ee5a913dc2a160c596 WatchSource:0}: Error finding container 609b45ad1520b8fd10f0be64867da3b734052a1df82c76ee5a913dc2a160c596: Status 404 returned error can't find the container with id 609b45ad1520b8fd10f0be64867da3b734052a1df82c76ee5a913dc2a160c596
Sep 17 18:35:45.477296 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:45.477262    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg" event={"ID":"a53f199e-e9ab-4d39-93b9-704fe1c6658a","Type":"ContainerStarted","Data":"a4ebc3be3ad9e7161135e3e4f2f0d6538ea3931ee01ab3f38c2a9bfdbc07acfd"}
Sep 17 18:35:45.477695 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:45.477341    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg" event={"ID":"a53f199e-e9ab-4d39-93b9-704fe1c6658a","Type":"ContainerStarted","Data":"609b45ad1520b8fd10f0be64867da3b734052a1df82c76ee5a913dc2a160c596"}
Sep 17 18:35:45.477695 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:45.477416    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg"
Sep 17 18:35:45.494846 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:35:45.494792    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg" podStartSLOduration=1.494778352 podStartE2EDuration="1.494778352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:35:45.49375768 +0000 UTC m=+2333.584379369" watchObservedRunningTime="2026-09-17 18:35:45.494778352 +0000 UTC m=+2333.585400002"
Sep 17 18:36:00.203417 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:00.203371    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"]
Sep 17 18:36:00.203838 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:00.203495    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl"]
Sep 17 18:36:06.865214 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:06.865173    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl" podUID="a8e1b2af-0b5a-44f4-a204-3c45e6ee64be" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:36:15.223596 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:15.223549    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl" podUID="a8e1b2af-0b5a-44f4-a204-3c45e6ee64be" containerName="main" containerID="cri-o://0f2438b2fd29abc07784db2a575ed9f15e644b5c4d3555c0e8bf4c4a3284c7a2" gracePeriod=45
Sep 17 18:36:15.234478 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:15.234438    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw" podUID="ffedf828-ea72-412c-bd99-6c1bacd01bf2" containerName="main" containerID="cri-o://6665327c20c794d71bfb12f9b266d3a659c6e1950784f69a3b34939244ef28ec" gracePeriod=45
Sep 17 18:36:15.875604 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:15.875575    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"
Sep 17 18:36:15.939790 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:15.939704    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffedf828-ea72-412c-bd99-6c1bacd01bf2-dshm\") pod \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\" (UID: \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\") "
Sep 17 18:36:15.939790 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:15.939764    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffedf828-ea72-412c-bd99-6c1bacd01bf2-tmp-dir\") pod \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\" (UID: \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\") "
Sep 17 18:36:15.940050 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:15.939804    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffedf828-ea72-412c-bd99-6c1bacd01bf2-kserve-provision-location\") pod \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\" (UID: \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\") "
Sep 17 18:36:15.940050 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:15.939849    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffedf828-ea72-412c-bd99-6c1bacd01bf2-home\") pod \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\" (UID: \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\") "
Sep 17 18:36:15.940050 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:15.939890    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffedf828-ea72-412c-bd99-6c1bacd01bf2-tls-certs\") pod \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\" (UID: \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\") "
Sep 17 18:36:15.940050 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:15.939934    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffedf828-ea72-412c-bd99-6c1bacd01bf2-cabundle-cert\") pod \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\" (UID: \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\") "
Sep 17 18:36:15.940050 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:15.939956    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffedf828-ea72-412c-bd99-6c1bacd01bf2-model-cache\") pod \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\" (UID: \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\") "
Sep 17 18:36:15.940050 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:15.940037    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtp4w\" (UniqueName: \"kubernetes.io/projected/ffedf828-ea72-412c-bd99-6c1bacd01bf2-kube-api-access-wtp4w\") pod \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\" (UID: \"ffedf828-ea72-412c-bd99-6c1bacd01bf2\") "
Sep 17 18:36:15.940395 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:15.940353    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffedf828-ea72-412c-bd99-6c1bacd01bf2-home" (OuterVolumeSpecName: "home") pod "ffedf828-ea72-412c-bd99-6c1bacd01bf2" (UID: "ffedf828-ea72-412c-bd99-6c1bacd01bf2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:36:15.940481 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:15.940397    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffedf828-ea72-412c-bd99-6c1bacd01bf2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ffedf828-ea72-412c-bd99-6c1bacd01bf2" (UID: "ffedf828-ea72-412c-bd99-6c1bacd01bf2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:36:15.940624 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:15.940587    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffedf828-ea72-412c-bd99-6c1bacd01bf2-model-cache" (OuterVolumeSpecName: "model-cache") pod "ffedf828-ea72-412c-bd99-6c1bacd01bf2" (UID: "ffedf828-ea72-412c-bd99-6c1bacd01bf2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:36:15.942830 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:15.942792    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffedf828-ea72-412c-bd99-6c1bacd01bf2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ffedf828-ea72-412c-bd99-6c1bacd01bf2" (UID: "ffedf828-ea72-412c-bd99-6c1bacd01bf2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:36:15.942961 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:15.942872    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffedf828-ea72-412c-bd99-6c1bacd01bf2-kube-api-access-wtp4w" (OuterVolumeSpecName: "kube-api-access-wtp4w") pod "ffedf828-ea72-412c-bd99-6c1bacd01bf2" (UID: "ffedf828-ea72-412c-bd99-6c1bacd01bf2"). InnerVolumeSpecName "kube-api-access-wtp4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:36:15.942961 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:15.942881    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffedf828-ea72-412c-bd99-6c1bacd01bf2-dshm" (OuterVolumeSpecName: "dshm") pod "ffedf828-ea72-412c-bd99-6c1bacd01bf2" (UID: "ffedf828-ea72-412c-bd99-6c1bacd01bf2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:36:15.952680 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:15.952652    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffedf828-ea72-412c-bd99-6c1bacd01bf2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ffedf828-ea72-412c-bd99-6c1bacd01bf2" (UID: "ffedf828-ea72-412c-bd99-6c1bacd01bf2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:36:15.965363 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:15.965324    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffedf828-ea72-412c-bd99-6c1bacd01bf2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ffedf828-ea72-412c-bd99-6c1bacd01bf2" (UID: "ffedf828-ea72-412c-bd99-6c1bacd01bf2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:36:16.041669 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:16.041632    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtp4w\" (UniqueName: \"kubernetes.io/projected/ffedf828-ea72-412c-bd99-6c1bacd01bf2-kube-api-access-wtp4w\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:36:16.041669 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:16.041663    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffedf828-ea72-412c-bd99-6c1bacd01bf2-dshm\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:36:16.041669 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:16.041673    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffedf828-ea72-412c-bd99-6c1bacd01bf2-tmp-dir\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:36:16.041883 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:16.041682    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffedf828-ea72-412c-bd99-6c1bacd01bf2-kserve-provision-location\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:36:16.041883 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:16.041691    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffedf828-ea72-412c-bd99-6c1bacd01bf2-home\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:36:16.041883 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:16.041699    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffedf828-ea72-412c-bd99-6c1bacd01bf2-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:36:16.041883 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:16.041707    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffedf828-ea72-412c-bd99-6c1bacd01bf2-cabundle-cert\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:36:16.041883 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:16.041716    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffedf828-ea72-412c-bd99-6c1bacd01bf2-model-cache\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:36:16.485211 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:16.485179    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg"
Sep 17 18:36:16.636761 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:16.636671    2581 generic.go:356] "Generic (PLEG): container finished" podID="ffedf828-ea72-412c-bd99-6c1bacd01bf2" containerID="6665327c20c794d71bfb12f9b266d3a659c6e1950784f69a3b34939244ef28ec" exitCode=0
Sep 17 18:36:16.636761 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:16.636732    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"
Sep 17 18:36:16.636960 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:16.636755    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw" event={"ID":"ffedf828-ea72-412c-bd99-6c1bacd01bf2","Type":"ContainerDied","Data":"6665327c20c794d71bfb12f9b266d3a659c6e1950784f69a3b34939244ef28ec"}
Sep 17 18:36:16.636960 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:16.636811    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw" event={"ID":"ffedf828-ea72-412c-bd99-6c1bacd01bf2","Type":"ContainerDied","Data":"bf53efdcddbaee3e65cb595c5348ca85c437324b3c01fa6e866cd9c7c08f98f7"}
Sep 17 18:36:16.636960 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:16.636833    2581 scope.go:117] "RemoveContainer" containerID="6665327c20c794d71bfb12f9b266d3a659c6e1950784f69a3b34939244ef28ec"
Sep 17 18:36:16.645225 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:16.645205    2581 scope.go:117] "RemoveContainer" containerID="f2aac0aa8750c3d0d10d15c18cb24c9d26bbe851700216062f389e7a4aa3176a"
Sep 17 18:36:16.653427 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:16.653409    2581 scope.go:117] "RemoveContainer" containerID="6665327c20c794d71bfb12f9b266d3a659c6e1950784f69a3b34939244ef28ec"
Sep 17 18:36:16.653774 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:36:16.653751    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6665327c20c794d71bfb12f9b266d3a659c6e1950784f69a3b34939244ef28ec\": container with ID starting with 6665327c20c794d71bfb12f9b266d3a659c6e1950784f69a3b34939244ef28ec not found: ID does not exist" containerID="6665327c20c794d71bfb12f9b266d3a659c6e1950784f69a3b34939244ef28ec"
Sep 17 18:36:16.653832 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:16.653778    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6665327c20c794d71bfb12f9b266d3a659c6e1950784f69a3b34939244ef28ec"} err="failed to get container status \"6665327c20c794d71bfb12f9b266d3a659c6e1950784f69a3b34939244ef28ec\": rpc error: code = NotFound desc = could not find container \"6665327c20c794d71bfb12f9b266d3a659c6e1950784f69a3b34939244ef28ec\": container with ID starting with 6665327c20c794d71bfb12f9b266d3a659c6e1950784f69a3b34939244ef28ec not found: ID does not exist"
Sep 17 18:36:16.653832 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:16.653794    2581 scope.go:117] "RemoveContainer" containerID="f2aac0aa8750c3d0d10d15c18cb24c9d26bbe851700216062f389e7a4aa3176a"
Sep 17 18:36:16.654092 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:36:16.654074    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2aac0aa8750c3d0d10d15c18cb24c9d26bbe851700216062f389e7a4aa3176a\": container with ID starting with f2aac0aa8750c3d0d10d15c18cb24c9d26bbe851700216062f389e7a4aa3176a not found: ID does not exist" containerID="f2aac0aa8750c3d0d10d15c18cb24c9d26bbe851700216062f389e7a4aa3176a"
Sep 17 18:36:16.654146 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:16.654097    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2aac0aa8750c3d0d10d15c18cb24c9d26bbe851700216062f389e7a4aa3176a"} err="failed to get container status \"f2aac0aa8750c3d0d10d15c18cb24c9d26bbe851700216062f389e7a4aa3176a\": rpc error: code = NotFound desc = could not find container \"f2aac0aa8750c3d0d10d15c18cb24c9d26bbe851700216062f389e7a4aa3176a\": container with ID starting with f2aac0aa8750c3d0d10d15c18cb24c9d26bbe851700216062f389e7a4aa3176a not found: ID does not exist"
Sep 17 18:36:16.654191 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:16.654174    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"]
Sep 17 18:36:16.657588 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:16.657564    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c4b8c95b7-9c5bw"]
Sep 17 18:36:16.864644 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:16.864607    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl" podUID="a8e1b2af-0b5a-44f4-a204-3c45e6ee64be" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:36:18.357635 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:18.357599    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffedf828-ea72-412c-bd99-6c1bacd01bf2" path="/var/lib/kubelet/pods/ffedf828-ea72-412c-bd99-6c1bacd01bf2/volumes"
Sep 17 18:36:26.864963 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:26.864860    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl" podUID="a8e1b2af-0b5a-44f4-a204-3c45e6ee64be" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:36:26.865438 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:26.864997    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl"
Sep 17 18:36:36.864950 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:36.864892    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl" podUID="a8e1b2af-0b5a-44f4-a204-3c45e6ee64be" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:36:45.469145 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:45.469113    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl"
Sep 17 18:36:45.497099 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:45.497025    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8e1b2af-0b5a-44f4-a204-3c45e6ee64be-tls-certs\") pod \"a8e1b2af-0b5a-44f4-a204-3c45e6ee64be\" (UID: \"a8e1b2af-0b5a-44f4-a204-3c45e6ee64be\") "
Sep 17 18:36:45.497239 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:45.497105    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cxj5\" (UniqueName: \"kubernetes.io/projected/a8e1b2af-0b5a-44f4-a204-3c45e6ee64be-kube-api-access-8cxj5\") pod \"a8e1b2af-0b5a-44f4-a204-3c45e6ee64be\" (UID: \"a8e1b2af-0b5a-44f4-a204-3c45e6ee64be\") "
Sep 17 18:36:45.499160 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:45.499122    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8e1b2af-0b5a-44f4-a204-3c45e6ee64be-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a8e1b2af-0b5a-44f4-a204-3c45e6ee64be" (UID: "a8e1b2af-0b5a-44f4-a204-3c45e6ee64be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:36:45.499279 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:45.499170    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e1b2af-0b5a-44f4-a204-3c45e6ee64be-kube-api-access-8cxj5" (OuterVolumeSpecName: "kube-api-access-8cxj5") pod "a8e1b2af-0b5a-44f4-a204-3c45e6ee64be" (UID: "a8e1b2af-0b5a-44f4-a204-3c45e6ee64be"). InnerVolumeSpecName "kube-api-access-8cxj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:36:45.598355 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:45.598300    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cxj5\" (UniqueName: \"kubernetes.io/projected/a8e1b2af-0b5a-44f4-a204-3c45e6ee64be-kube-api-access-8cxj5\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:36:45.598355 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:45.598348    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8e1b2af-0b5a-44f4-a204-3c45e6ee64be-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:36:45.770515 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:45.770425    2581 generic.go:356] "Generic (PLEG): container finished" podID="a8e1b2af-0b5a-44f4-a204-3c45e6ee64be" containerID="0f2438b2fd29abc07784db2a575ed9f15e644b5c4d3555c0e8bf4c4a3284c7a2" exitCode=0
Sep 17 18:36:45.770515 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:45.770470    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl"
Sep 17 18:36:45.770697 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:45.770515    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl" event={"ID":"a8e1b2af-0b5a-44f4-a204-3c45e6ee64be","Type":"ContainerDied","Data":"0f2438b2fd29abc07784db2a575ed9f15e644b5c4d3555c0e8bf4c4a3284c7a2"}
Sep 17 18:36:45.770697 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:45.770565    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl" event={"ID":"a8e1b2af-0b5a-44f4-a204-3c45e6ee64be","Type":"ContainerDied","Data":"8067833c135b5e86186ee6f2a27fbbd7abd80266b705c49a4b77cfc030a53159"}
Sep 17 18:36:45.770697 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:45.770588    2581 scope.go:117] "RemoveContainer" containerID="0f2438b2fd29abc07784db2a575ed9f15e644b5c4d3555c0e8bf4c4a3284c7a2"
Sep 17 18:36:45.780079 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:45.780062    2581 scope.go:117] "RemoveContainer" containerID="0f2438b2fd29abc07784db2a575ed9f15e644b5c4d3555c0e8bf4c4a3284c7a2"
Sep 17 18:36:45.780342 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:36:45.780322    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f2438b2fd29abc07784db2a575ed9f15e644b5c4d3555c0e8bf4c4a3284c7a2\": container with ID starting with 0f2438b2fd29abc07784db2a575ed9f15e644b5c4d3555c0e8bf4c4a3284c7a2 not found: ID does not exist" containerID="0f2438b2fd29abc07784db2a575ed9f15e644b5c4d3555c0e8bf4c4a3284c7a2"
Sep 17 18:36:45.780390 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:45.780350    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f2438b2fd29abc07784db2a575ed9f15e644b5c4d3555c0e8bf4c4a3284c7a2"} err="failed to get container status \"0f2438b2fd29abc07784db2a575ed9f15e644b5c4d3555c0e8bf4c4a3284c7a2\": rpc error: code = NotFound desc = could not find container \"0f2438b2fd29abc07784db2a575ed9f15e644b5c4d3555c0e8bf4c4a3284c7a2\": container with ID starting with 0f2438b2fd29abc07784db2a575ed9f15e644b5c4d3555c0e8bf4c4a3284c7a2 not found: ID does not exist"
Sep 17 18:36:45.791639 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:45.791611    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl"]
Sep 17 18:36:45.794967 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:45.794944    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche8qqnl"]
Sep 17 18:36:46.358345 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:46.358311    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8e1b2af-0b5a-44f4-a204-3c45e6ee64be" path="/var/lib/kubelet/pods/a8e1b2af-0b5a-44f4-a204-3c45e6ee64be/volumes"
Sep 17 18:36:52.533937 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:52.533884    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/2.log"
Sep 17 18:36:52.545194 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:36:52.545171    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/2.log"
Sep 17 18:37:05.172866 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.172833    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"]
Sep 17 18:37:05.173430 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.173301    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffedf828-ea72-412c-bd99-6c1bacd01bf2" containerName="main"
Sep 17 18:37:05.173430 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.173322    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffedf828-ea72-412c-bd99-6c1bacd01bf2" containerName="main"
Sep 17 18:37:05.173430 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.173344    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffedf828-ea72-412c-bd99-6c1bacd01bf2" containerName="storage-initializer"
Sep 17 18:37:05.173430 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.173354    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffedf828-ea72-412c-bd99-6c1bacd01bf2" containerName="storage-initializer"
Sep 17 18:37:05.173430 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.173416    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8e1b2af-0b5a-44f4-a204-3c45e6ee64be" containerName="main"
Sep 17 18:37:05.173430 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.173428    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8e1b2af-0b5a-44f4-a204-3c45e6ee64be" containerName="main"
Sep 17 18:37:05.173708 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.173529    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffedf828-ea72-412c-bd99-6c1bacd01bf2" containerName="main"
Sep 17 18:37:05.173708 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.173545    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8e1b2af-0b5a-44f4-a204-3c45e6ee64be" containerName="main"
Sep 17 18:37:05.176573 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.176553    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"
Sep 17 18:37:05.179418 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.179394    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:37:05.179573 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.179549    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-qnzgf\""
Sep 17 18:37:05.179655 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.179449    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 17 18:37:05.179787 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.179769    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Sep 17 18:37:05.179872 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.179775    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 17 18:37:05.179966 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.179786    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 17 18:37:05.192830 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.192802    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"]
Sep 17 18:37:05.278001 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.277964    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7sjf\" (UniqueName: \"kubernetes.io/projected/9647927d-3657-4748-939a-0014a632ffe5-kube-api-access-m7sjf\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll\" (UID: \"9647927d-3657-4748-939a-0014a632ffe5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"
Sep 17 18:37:05.278176 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.278013    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9647927d-3657-4748-939a-0014a632ffe5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll\" (UID: \"9647927d-3657-4748-939a-0014a632ffe5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"
Sep 17 18:37:05.278176 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.278075    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9647927d-3657-4748-939a-0014a632ffe5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll\" (UID: \"9647927d-3657-4748-939a-0014a632ffe5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"
Sep 17 18:37:05.278176 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.278145    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9647927d-3657-4748-939a-0014a632ffe5-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll\" (UID: \"9647927d-3657-4748-939a-0014a632ffe5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"
Sep 17 18:37:05.278286 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.278191    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9647927d-3657-4748-939a-0014a632ffe5-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll\" (UID: \"9647927d-3657-4748-939a-0014a632ffe5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"
Sep 17 18:37:05.278286 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.278217    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9647927d-3657-4748-939a-0014a632ffe5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll\" (UID: \"9647927d-3657-4748-939a-0014a632ffe5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"
Sep 17 18:37:05.278286 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.278273    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9647927d-3657-4748-939a-0014a632ffe5-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll\" (UID: \"9647927d-3657-4748-939a-0014a632ffe5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"
Sep 17 18:37:05.278383 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.278295    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9647927d-3657-4748-939a-0014a632ffe5-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll\" (UID: \"9647927d-3657-4748-939a-0014a632ffe5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"
Sep 17 18:37:05.379471 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.379428    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9647927d-3657-4748-939a-0014a632ffe5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll\" (UID: \"9647927d-3657-4748-939a-0014a632ffe5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"
Sep 17 18:37:05.379658 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.379499    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9647927d-3657-4748-939a-0014a632ffe5-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll\" (UID: \"9647927d-3657-4748-939a-0014a632ffe5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"
Sep 17 18:37:05.379704 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.379656    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9647927d-3657-4748-939a-0014a632ffe5-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll\" (UID: \"9647927d-3657-4748-939a-0014a632ffe5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"
Sep 17 18:37:05.379751 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.379700    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9647927d-3657-4748-939a-0014a632ffe5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll\" (UID: \"9647927d-3657-4748-939a-0014a632ffe5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"
Sep 17 18:37:05.379751 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.379728    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9647927d-3657-4748-939a-0014a632ffe5-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll\" (UID: \"9647927d-3657-4748-939a-0014a632ffe5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"
Sep 17 18:37:05.379849 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.379757    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9647927d-3657-4748-939a-0014a632ffe5-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll\" (UID: \"9647927d-3657-4748-939a-0014a632ffe5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"
Sep 17 18:37:05.379849 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.379811    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7sjf\" (UniqueName: \"kubernetes.io/projected/9647927d-3657-4748-939a-0014a632ffe5-kube-api-access-m7sjf\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll\" (UID: \"9647927d-3657-4748-939a-0014a632ffe5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"
Sep 17 18:37:05.379849 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.379842    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9647927d-3657-4748-939a-0014a632ffe5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll\" (UID: \"9647927d-3657-4748-939a-0014a632ffe5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"
Sep 17 18:37:05.380056 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.379945    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9647927d-3657-4748-939a-0014a632ffe5-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll\" (UID: \"9647927d-3657-4748-939a-0014a632ffe5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"
Sep 17 18:37:05.380220 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.380186    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9647927d-3657-4748-939a-0014a632ffe5-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll\" (UID: \"9647927d-3657-4748-939a-0014a632ffe5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"
Sep 17 18:37:05.380310 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.380256    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9647927d-3657-4748-939a-0014a632ffe5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll\" (UID: \"9647927d-3657-4748-939a-0014a632ffe5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"
Sep 17 18:37:05.380422 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.380399    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9647927d-3657-4748-939a-0014a632ffe5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll\" (UID: \"9647927d-3657-4748-939a-0014a632ffe5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"
Sep 17 18:37:05.380595 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.380495    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9647927d-3657-4748-939a-0014a632ffe5-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll\" (UID: \"9647927d-3657-4748-939a-0014a632ffe5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"
Sep 17 18:37:05.381885 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.381868    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9647927d-3657-4748-939a-0014a632ffe5-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll\" (UID: \"9647927d-3657-4748-939a-0014a632ffe5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"
Sep 17 18:37:05.382173 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.382155    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9647927d-3657-4748-939a-0014a632ffe5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll\" (UID: \"9647927d-3657-4748-939a-0014a632ffe5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"
Sep 17 18:37:05.387695 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.387671    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7sjf\" (UniqueName: \"kubernetes.io/projected/9647927d-3657-4748-939a-0014a632ffe5-kube-api-access-m7sjf\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll\" (UID: \"9647927d-3657-4748-939a-0014a632ffe5\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"
Sep 17 18:37:05.488076 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.488030    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"
Sep 17 18:37:05.765827 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.765799    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"]
Sep 17 18:37:05.768278 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:37:05.768245    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9647927d_3657_4748_939a_0014a632ffe5.slice/crio-00c39ca132bb852ad99343ef079d5902f55fff081c8bb8374b798178d3e61092 WatchSource:0}: Error finding container 00c39ca132bb852ad99343ef079d5902f55fff081c8bb8374b798178d3e61092: Status 404 returned error can't find the container with id 00c39ca132bb852ad99343ef079d5902f55fff081c8bb8374b798178d3e61092
Sep 17 18:37:05.865768 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.865726    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll" event={"ID":"9647927d-3657-4748-939a-0014a632ffe5","Type":"ContainerStarted","Data":"133865205f0c3c25884a348509cb614b66b02634e2875100835c6f6da02edc30"}
Sep 17 18:37:05.866009 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:05.865788    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll" event={"ID":"9647927d-3657-4748-939a-0014a632ffe5","Type":"ContainerStarted","Data":"00c39ca132bb852ad99343ef079d5902f55fff081c8bb8374b798178d3e61092"}
Sep 17 18:37:08.881248 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:08.881147    2581 generic.go:356] "Generic (PLEG): container finished" podID="9647927d-3657-4748-939a-0014a632ffe5" containerID="133865205f0c3c25884a348509cb614b66b02634e2875100835c6f6da02edc30" exitCode=0
Sep 17 18:37:08.881248 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:08.881220    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll" event={"ID":"9647927d-3657-4748-939a-0014a632ffe5","Type":"ContainerDied","Data":"133865205f0c3c25884a348509cb614b66b02634e2875100835c6f6da02edc30"}
Sep 17 18:37:09.888169 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:09.888132    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll" event={"ID":"9647927d-3657-4748-939a-0014a632ffe5","Type":"ContainerStarted","Data":"59891ada7773839d23fe683c74c2972839e393cfdf187a253316b5e7e98ab136"}
Sep 17 18:37:09.911497 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:09.911435    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll" podStartSLOduration=2.114559435 podStartE2EDuration="4.911418452s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.796859017s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:37:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:37:09.908588528 +0000 UTC m=+2417.999210179" watchObservedRunningTime="2026-09-17 18:37:09.911418452 +0000 UTC m=+2418.002040101"
Sep 17 18:37:10.488989 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:10.488957    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"
Sep 17 18:37:15.489100 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:15.489008    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"
Sep 17 18:37:15.490517 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:15.490488    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll" podUID="9647927d-3657-4748-939a-0014a632ffe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 17 18:37:25.488596 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:25.488549    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll" podUID="9647927d-3657-4748-939a-0014a632ffe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 17 18:37:35.489519 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:35.489465    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll" podUID="9647927d-3657-4748-939a-0014a632ffe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 17 18:37:45.489301 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:45.489254    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll" podUID="9647927d-3657-4748-939a-0014a632ffe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 17 18:37:48.807060 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:48.807021    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg"]
Sep 17 18:37:55.488801 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:55.488751    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll" podUID="9647927d-3657-4748-939a-0014a632ffe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 17 18:37:56.484503 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:37:56.484461    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg" podUID="a53f199e-e9ab-4d39-93b9-704fe1c6658a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:38:03.827533 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:03.827481    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg" podUID="a53f199e-e9ab-4d39-93b9-704fe1c6658a" containerName="main" containerID="cri-o://a4ebc3be3ad9e7161135e3e4f2f0d6538ea3931ee01ab3f38c2a9bfdbc07acfd" gracePeriod=45
Sep 17 18:38:05.489112 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:05.489057    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll" podUID="9647927d-3657-4748-939a-0014a632ffe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 17 18:38:06.484471 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:06.484430    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg" podUID="a53f199e-e9ab-4d39-93b9-704fe1c6658a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:38:15.488729 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:15.488685    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll" podUID="9647927d-3657-4748-939a-0014a632ffe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 17 18:38:16.484354 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:16.484317    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg" podUID="a53f199e-e9ab-4d39-93b9-704fe1c6658a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:38:16.484526 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:16.484427    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg"
Sep 17 18:38:25.499279 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:25.499244    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"
Sep 17 18:38:25.507071 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:25.507040    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"
Sep 17 18:38:26.484997 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:26.484950    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg" podUID="a53f199e-e9ab-4d39-93b9-704fe1c6658a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:38:27.228487 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:27.228447    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"]
Sep 17 18:38:34.079202 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:34.079173    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg"
Sep 17 18:38:34.203309 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:34.203283    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xt49l\" (UniqueName: \"kubernetes.io/projected/a53f199e-e9ab-4d39-93b9-704fe1c6658a-kube-api-access-xt49l\") pod \"a53f199e-e9ab-4d39-93b9-704fe1c6658a\" (UID: \"a53f199e-e9ab-4d39-93b9-704fe1c6658a\") "
Sep 17 18:38:34.203479 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:34.203361    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a53f199e-e9ab-4d39-93b9-704fe1c6658a-tls-certs\") pod \"a53f199e-e9ab-4d39-93b9-704fe1c6658a\" (UID: \"a53f199e-e9ab-4d39-93b9-704fe1c6658a\") "
Sep 17 18:38:34.205554 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:34.205524    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a53f199e-e9ab-4d39-93b9-704fe1c6658a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a53f199e-e9ab-4d39-93b9-704fe1c6658a" (UID: "a53f199e-e9ab-4d39-93b9-704fe1c6658a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:38:34.205554 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:34.205537    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53f199e-e9ab-4d39-93b9-704fe1c6658a-kube-api-access-xt49l" (OuterVolumeSpecName: "kube-api-access-xt49l") pod "a53f199e-e9ab-4d39-93b9-704fe1c6658a" (UID: "a53f199e-e9ab-4d39-93b9-704fe1c6658a"). InnerVolumeSpecName "kube-api-access-xt49l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:38:34.305000 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:34.304942    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt49l\" (UniqueName: \"kubernetes.io/projected/a53f199e-e9ab-4d39-93b9-704fe1c6658a-kube-api-access-xt49l\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:38:34.305000 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:34.304998    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a53f199e-e9ab-4d39-93b9-704fe1c6658a-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:38:34.321082 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:34.321045    2581 generic.go:356] "Generic (PLEG): container finished" podID="a53f199e-e9ab-4d39-93b9-704fe1c6658a" containerID="a4ebc3be3ad9e7161135e3e4f2f0d6538ea3931ee01ab3f38c2a9bfdbc07acfd" exitCode=0
Sep 17 18:38:34.321082 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:34.321075    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg" event={"ID":"a53f199e-e9ab-4d39-93b9-704fe1c6658a","Type":"ContainerDied","Data":"a4ebc3be3ad9e7161135e3e4f2f0d6538ea3931ee01ab3f38c2a9bfdbc07acfd"}
Sep 17 18:38:34.321281 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:34.321092    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg"
Sep 17 18:38:34.321281 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:34.321130    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg" event={"ID":"a53f199e-e9ab-4d39-93b9-704fe1c6658a","Type":"ContainerDied","Data":"609b45ad1520b8fd10f0be64867da3b734052a1df82c76ee5a913dc2a160c596"}
Sep 17 18:38:34.321281 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:34.321153    2581 scope.go:117] "RemoveContainer" containerID="a4ebc3be3ad9e7161135e3e4f2f0d6538ea3931ee01ab3f38c2a9bfdbc07acfd"
Sep 17 18:38:34.330596 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:34.330577    2581 scope.go:117] "RemoveContainer" containerID="a4ebc3be3ad9e7161135e3e4f2f0d6538ea3931ee01ab3f38c2a9bfdbc07acfd"
Sep 17 18:38:34.330845 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:38:34.330820    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4ebc3be3ad9e7161135e3e4f2f0d6538ea3931ee01ab3f38c2a9bfdbc07acfd\": container with ID starting with a4ebc3be3ad9e7161135e3e4f2f0d6538ea3931ee01ab3f38c2a9bfdbc07acfd not found: ID does not exist" containerID="a4ebc3be3ad9e7161135e3e4f2f0d6538ea3931ee01ab3f38c2a9bfdbc07acfd"
Sep 17 18:38:34.330942 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:34.330856    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4ebc3be3ad9e7161135e3e4f2f0d6538ea3931ee01ab3f38c2a9bfdbc07acfd"} err="failed to get container status \"a4ebc3be3ad9e7161135e3e4f2f0d6538ea3931ee01ab3f38c2a9bfdbc07acfd\": rpc error: code = NotFound desc = could not find container \"a4ebc3be3ad9e7161135e3e4f2f0d6538ea3931ee01ab3f38c2a9bfdbc07acfd\": container with ID starting with a4ebc3be3ad9e7161135e3e4f2f0d6538ea3931ee01ab3f38c2a9bfdbc07acfd not found: ID does not exist"
Sep 17 18:38:34.342681 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:34.342659    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg"]
Sep 17 18:38:34.348297 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:34.348274    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc5888dhg"]
Sep 17 18:38:34.357303 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:34.357278    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a53f199e-e9ab-4d39-93b9-704fe1c6658a" path="/var/lib/kubelet/pods/a53f199e-e9ab-4d39-93b9-704fe1c6658a/volumes"
Sep 17 18:38:39.144525 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.144489    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"]
Sep 17 18:38:39.145059 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.144862    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a53f199e-e9ab-4d39-93b9-704fe1c6658a" containerName="main"
Sep 17 18:38:39.145059 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.144874    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a53f199e-e9ab-4d39-93b9-704fe1c6658a" containerName="main"
Sep 17 18:38:39.145059 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.144947    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a53f199e-e9ab-4d39-93b9-704fe1c6658a" containerName="main"
Sep 17 18:38:39.149429 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.149407    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:39.153164 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.153138    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 17 18:38:39.153271 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.153207    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Sep 17 18:38:39.153390 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.153358    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-zklvt\""
Sep 17 18:38:39.154284 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.154263    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 17 18:38:39.159443 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.159421    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"]
Sep 17 18:38:39.245134 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.245099    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s2x7\" (UniqueName: \"kubernetes.io/projected/47140786-7812-4af5-85c6-5528ce2bdd3a-kube-api-access-2s2x7\") pod \"router-gateway-1-openshift-default-cc7cb45dd-mcftx\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:39.245293 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.245139    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47140786-7812-4af5-85c6-5528ce2bdd3a-credential-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-mcftx\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:39.245293 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.245166    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47140786-7812-4af5-85c6-5528ce2bdd3a-workload-certs\") pod \"router-gateway-1-openshift-default-cc7cb45dd-mcftx\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:39.245293 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.245234    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47140786-7812-4af5-85c6-5528ce2bdd3a-istio-envoy\") pod \"router-gateway-1-openshift-default-cc7cb45dd-mcftx\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:39.245435 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.245290    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47140786-7812-4af5-85c6-5528ce2bdd3a-istio-token\") pod \"router-gateway-1-openshift-default-cc7cb45dd-mcftx\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:39.245435 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.245348    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47140786-7812-4af5-85c6-5528ce2bdd3a-workload-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-mcftx\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:39.245435 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.245371    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47140786-7812-4af5-85c6-5528ce2bdd3a-istio-podinfo\") pod \"router-gateway-1-openshift-default-cc7cb45dd-mcftx\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:39.245591 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.245439    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47140786-7812-4af5-85c6-5528ce2bdd3a-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-cc7cb45dd-mcftx\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:39.245591 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.245469    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47140786-7812-4af5-85c6-5528ce2bdd3a-istio-data\") pod \"router-gateway-1-openshift-default-cc7cb45dd-mcftx\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:39.346328 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.346294    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47140786-7812-4af5-85c6-5528ce2bdd3a-workload-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-mcftx\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:39.346328 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.346330    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47140786-7812-4af5-85c6-5528ce2bdd3a-istio-podinfo\") pod \"router-gateway-1-openshift-default-cc7cb45dd-mcftx\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:39.346568 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.346352    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47140786-7812-4af5-85c6-5528ce2bdd3a-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-cc7cb45dd-mcftx\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:39.346568 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.346371    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47140786-7812-4af5-85c6-5528ce2bdd3a-istio-data\") pod \"router-gateway-1-openshift-default-cc7cb45dd-mcftx\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:39.346568 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.346406    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s2x7\" (UniqueName: \"kubernetes.io/projected/47140786-7812-4af5-85c6-5528ce2bdd3a-kube-api-access-2s2x7\") pod \"router-gateway-1-openshift-default-cc7cb45dd-mcftx\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:39.346568 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.346434    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47140786-7812-4af5-85c6-5528ce2bdd3a-credential-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-mcftx\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:39.346568 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.346458    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47140786-7812-4af5-85c6-5528ce2bdd3a-workload-certs\") pod \"router-gateway-1-openshift-default-cc7cb45dd-mcftx\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:39.346568 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.346509    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47140786-7812-4af5-85c6-5528ce2bdd3a-istio-envoy\") pod \"router-gateway-1-openshift-default-cc7cb45dd-mcftx\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:39.346568 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.346556    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47140786-7812-4af5-85c6-5528ce2bdd3a-istio-token\") pod \"router-gateway-1-openshift-default-cc7cb45dd-mcftx\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:39.346971 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.346768    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47140786-7812-4af5-85c6-5528ce2bdd3a-workload-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-mcftx\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:39.346971 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.346894    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47140786-7812-4af5-85c6-5528ce2bdd3a-credential-socket\") pod \"router-gateway-1-openshift-default-cc7cb45dd-mcftx\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:39.347161 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.347140    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47140786-7812-4af5-85c6-5528ce2bdd3a-workload-certs\") pod \"router-gateway-1-openshift-default-cc7cb45dd-mcftx\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:39.347245 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.347161    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47140786-7812-4af5-85c6-5528ce2bdd3a-istio-data\") pod \"router-gateway-1-openshift-default-cc7cb45dd-mcftx\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:39.347591 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.347568    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47140786-7812-4af5-85c6-5528ce2bdd3a-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-cc7cb45dd-mcftx\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:39.349059 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.349035    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47140786-7812-4af5-85c6-5528ce2bdd3a-istio-envoy\") pod \"router-gateway-1-openshift-default-cc7cb45dd-mcftx\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:39.349138 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.349124    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47140786-7812-4af5-85c6-5528ce2bdd3a-istio-podinfo\") pod \"router-gateway-1-openshift-default-cc7cb45dd-mcftx\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:39.354468 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.354444    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47140786-7812-4af5-85c6-5528ce2bdd3a-istio-token\") pod \"router-gateway-1-openshift-default-cc7cb45dd-mcftx\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:39.355234 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.355211    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s2x7\" (UniqueName: \"kubernetes.io/projected/47140786-7812-4af5-85c6-5528ce2bdd3a-kube-api-access-2s2x7\") pod \"router-gateway-1-openshift-default-cc7cb45dd-mcftx\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:39.462731 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.462699    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:39.741775 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.741749    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"]
Sep 17 18:38:39.744134 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:38:39.744104    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47140786_7812_4af5_85c6_5528ce2bdd3a.slice/crio-5cc3c5c27c6022eb017cdeff7e6aafbda77f2661fa3e1ed1c9b700b1b39e40b2 WatchSource:0}: Error finding container 5cc3c5c27c6022eb017cdeff7e6aafbda77f2661fa3e1ed1c9b700b1b39e40b2: Status 404 returned error can't find the container with id 5cc3c5c27c6022eb017cdeff7e6aafbda77f2661fa3e1ed1c9b700b1b39e40b2
Sep 17 18:38:39.746133 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.746100    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:38:39.746228 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.746166    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:38:39.746228 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:39.746192    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:38:40.357792 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:40.357755    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx" event={"ID":"47140786-7812-4af5-85c6-5528ce2bdd3a","Type":"ContainerStarted","Data":"6d6782d230ec4bbfea00e673ee075729176d147fc746217d52abb8c615fa817e"}
Sep 17 18:38:40.358161 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:40.357812    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx" event={"ID":"47140786-7812-4af5-85c6-5528ce2bdd3a","Type":"ContainerStarted","Data":"5cc3c5c27c6022eb017cdeff7e6aafbda77f2661fa3e1ed1c9b700b1b39e40b2"}
Sep 17 18:38:40.380668 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:40.380616    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx" podStartSLOduration=1.380600498 podStartE2EDuration="1.380600498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:38:40.378740695 +0000 UTC m=+2508.469362368" watchObservedRunningTime="2026-09-17 18:38:40.380600498 +0000 UTC m=+2508.471222149"
Sep 17 18:38:40.463875 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:40.463844    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:40.468879 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:40.468855    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:41.361786 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:41.361755    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:41.406958 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:41.406897    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:42.314835 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:42.314651    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll" podUID="9647927d-3657-4748-939a-0014a632ffe5" containerName="main" containerID="cri-o://59891ada7773839d23fe683c74c2972839e393cfdf187a253316b5e7e98ab136" gracePeriod=45
Sep 17 18:38:42.499863 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:42.499826    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll" podUID="9647927d-3657-4748-939a-0014a632ffe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 17 18:38:42.985361 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:42.985336    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"
Sep 17 18:38:43.083191 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.083161    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7sjf\" (UniqueName: \"kubernetes.io/projected/9647927d-3657-4748-939a-0014a632ffe5-kube-api-access-m7sjf\") pod \"9647927d-3657-4748-939a-0014a632ffe5\" (UID: \"9647927d-3657-4748-939a-0014a632ffe5\") "
Sep 17 18:38:43.083191 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.083198    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9647927d-3657-4748-939a-0014a632ffe5-tmp-dir\") pod \"9647927d-3657-4748-939a-0014a632ffe5\" (UID: \"9647927d-3657-4748-939a-0014a632ffe5\") "
Sep 17 18:38:43.083418 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.083221    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9647927d-3657-4748-939a-0014a632ffe5-model-cache\") pod \"9647927d-3657-4748-939a-0014a632ffe5\" (UID: \"9647927d-3657-4748-939a-0014a632ffe5\") "
Sep 17 18:38:43.083418 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.083242    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9647927d-3657-4748-939a-0014a632ffe5-tls-certs\") pod \"9647927d-3657-4748-939a-0014a632ffe5\" (UID: \"9647927d-3657-4748-939a-0014a632ffe5\") "
Sep 17 18:38:43.083418 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.083293    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9647927d-3657-4748-939a-0014a632ffe5-cabundle-cert\") pod \"9647927d-3657-4748-939a-0014a632ffe5\" (UID: \"9647927d-3657-4748-939a-0014a632ffe5\") "
Sep 17 18:38:43.083418 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.083324    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9647927d-3657-4748-939a-0014a632ffe5-home\") pod \"9647927d-3657-4748-939a-0014a632ffe5\" (UID: \"9647927d-3657-4748-939a-0014a632ffe5\") "
Sep 17 18:38:43.083418 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.083367    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9647927d-3657-4748-939a-0014a632ffe5-kserve-provision-location\") pod \"9647927d-3657-4748-939a-0014a632ffe5\" (UID: \"9647927d-3657-4748-939a-0014a632ffe5\") "
Sep 17 18:38:43.083418 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.083394    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9647927d-3657-4748-939a-0014a632ffe5-dshm\") pod \"9647927d-3657-4748-939a-0014a632ffe5\" (UID: \"9647927d-3657-4748-939a-0014a632ffe5\") "
Sep 17 18:38:43.083727 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.083507    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9647927d-3657-4748-939a-0014a632ffe5-model-cache" (OuterVolumeSpecName: "model-cache") pod "9647927d-3657-4748-939a-0014a632ffe5" (UID: "9647927d-3657-4748-939a-0014a632ffe5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:38:43.083790 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.083733    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9647927d-3657-4748-939a-0014a632ffe5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9647927d-3657-4748-939a-0014a632ffe5" (UID: "9647927d-3657-4748-939a-0014a632ffe5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:38:43.083790 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.083748    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9647927d-3657-4748-939a-0014a632ffe5-model-cache\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:38:43.083790 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.083758    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9647927d-3657-4748-939a-0014a632ffe5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9647927d-3657-4748-939a-0014a632ffe5" (UID: "9647927d-3657-4748-939a-0014a632ffe5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:38:43.083958 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.083803    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9647927d-3657-4748-939a-0014a632ffe5-home" (OuterVolumeSpecName: "home") pod "9647927d-3657-4748-939a-0014a632ffe5" (UID: "9647927d-3657-4748-939a-0014a632ffe5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:38:43.085656 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.085630    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9647927d-3657-4748-939a-0014a632ffe5-dshm" (OuterVolumeSpecName: "dshm") pod "9647927d-3657-4748-939a-0014a632ffe5" (UID: "9647927d-3657-4748-939a-0014a632ffe5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:38:43.085796 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.085692    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9647927d-3657-4748-939a-0014a632ffe5-kube-api-access-m7sjf" (OuterVolumeSpecName: "kube-api-access-m7sjf") pod "9647927d-3657-4748-939a-0014a632ffe5" (UID: "9647927d-3657-4748-939a-0014a632ffe5"). InnerVolumeSpecName "kube-api-access-m7sjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:38:43.085796 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.085748    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9647927d-3657-4748-939a-0014a632ffe5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9647927d-3657-4748-939a-0014a632ffe5" (UID: "9647927d-3657-4748-939a-0014a632ffe5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:38:43.125927 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.125075    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9647927d-3657-4748-939a-0014a632ffe5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9647927d-3657-4748-939a-0014a632ffe5" (UID: "9647927d-3657-4748-939a-0014a632ffe5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:38:43.184725 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.184629    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9647927d-3657-4748-939a-0014a632ffe5-home\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:38:43.184725 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.184665    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9647927d-3657-4748-939a-0014a632ffe5-kserve-provision-location\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:38:43.184725 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.184678    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9647927d-3657-4748-939a-0014a632ffe5-dshm\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:38:43.184725 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.184691    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7sjf\" (UniqueName: \"kubernetes.io/projected/9647927d-3657-4748-939a-0014a632ffe5-kube-api-access-m7sjf\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:38:43.184725 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.184703    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9647927d-3657-4748-939a-0014a632ffe5-tmp-dir\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:38:43.184725 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.184714    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9647927d-3657-4748-939a-0014a632ffe5-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:38:43.184725 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.184725    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9647927d-3657-4748-939a-0014a632ffe5-cabundle-cert\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:38:43.371931 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.371871    2581 generic.go:356] "Generic (PLEG): container finished" podID="9647927d-3657-4748-939a-0014a632ffe5" containerID="59891ada7773839d23fe683c74c2972839e393cfdf187a253316b5e7e98ab136" exitCode=0
Sep 17 18:38:43.372097 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.371953    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"
Sep 17 18:38:43.372097 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.371957    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll" event={"ID":"9647927d-3657-4748-939a-0014a632ffe5","Type":"ContainerDied","Data":"59891ada7773839d23fe683c74c2972839e393cfdf187a253316b5e7e98ab136"}
Sep 17 18:38:43.372097 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.372005    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll" event={"ID":"9647927d-3657-4748-939a-0014a632ffe5","Type":"ContainerDied","Data":"00c39ca132bb852ad99343ef079d5902f55fff081c8bb8374b798178d3e61092"}
Sep 17 18:38:43.372097 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.372027    2581 scope.go:117] "RemoveContainer" containerID="59891ada7773839d23fe683c74c2972839e393cfdf187a253316b5e7e98ab136"
Sep 17 18:38:43.381568 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.381536    2581 scope.go:117] "RemoveContainer" containerID="133865205f0c3c25884a348509cb614b66b02634e2875100835c6f6da02edc30"
Sep 17 18:38:43.390364 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.390339    2581 scope.go:117] "RemoveContainer" containerID="59891ada7773839d23fe683c74c2972839e393cfdf187a253316b5e7e98ab136"
Sep 17 18:38:43.390673 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:38:43.390647    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59891ada7773839d23fe683c74c2972839e393cfdf187a253316b5e7e98ab136\": container with ID starting with 59891ada7773839d23fe683c74c2972839e393cfdf187a253316b5e7e98ab136 not found: ID does not exist" containerID="59891ada7773839d23fe683c74c2972839e393cfdf187a253316b5e7e98ab136"
Sep 17 18:38:43.390756 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.390688    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59891ada7773839d23fe683c74c2972839e393cfdf187a253316b5e7e98ab136"} err="failed to get container status \"59891ada7773839d23fe683c74c2972839e393cfdf187a253316b5e7e98ab136\": rpc error: code = NotFound desc = could not find container \"59891ada7773839d23fe683c74c2972839e393cfdf187a253316b5e7e98ab136\": container with ID starting with 59891ada7773839d23fe683c74c2972839e393cfdf187a253316b5e7e98ab136 not found: ID does not exist"
Sep 17 18:38:43.390756 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.390715    2581 scope.go:117] "RemoveContainer" containerID="133865205f0c3c25884a348509cb614b66b02634e2875100835c6f6da02edc30"
Sep 17 18:38:43.391073 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:38:43.391052    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"133865205f0c3c25884a348509cb614b66b02634e2875100835c6f6da02edc30\": container with ID starting with 133865205f0c3c25884a348509cb614b66b02634e2875100835c6f6da02edc30 not found: ID does not exist" containerID="133865205f0c3c25884a348509cb614b66b02634e2875100835c6f6da02edc30"
Sep 17 18:38:43.391144 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.391079    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"133865205f0c3c25884a348509cb614b66b02634e2875100835c6f6da02edc30"} err="failed to get container status \"133865205f0c3c25884a348509cb614b66b02634e2875100835c6f6da02edc30\": rpc error: code = NotFound desc = could not find container \"133865205f0c3c25884a348509cb614b66b02634e2875100835c6f6da02edc30\": container with ID starting with 133865205f0c3c25884a348509cb614b66b02634e2875100835c6f6da02edc30 not found: ID does not exist"
Sep 17 18:38:43.394967 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.394946    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"]
Sep 17 18:38:43.398889 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:43.398869    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-dbd856dc6-2bwll"]
Sep 17 18:38:44.358042 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:44.358005    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9647927d-3657-4748-939a-0014a632ffe5" path="/var/lib/kubelet/pods/9647927d-3657-4748-939a-0014a632ffe5/volumes"
Sep 17 18:38:45.080024 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.079940    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"]
Sep 17 18:38:45.080436 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.080415    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9647927d-3657-4748-939a-0014a632ffe5" containerName="main"
Sep 17 18:38:45.080504 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.080439    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9647927d-3657-4748-939a-0014a632ffe5" containerName="main"
Sep 17 18:38:45.080504 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.080467    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9647927d-3657-4748-939a-0014a632ffe5" containerName="storage-initializer"
Sep 17 18:38:45.080504 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.080474    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9647927d-3657-4748-939a-0014a632ffe5" containerName="storage-initializer"
Sep 17 18:38:45.080620 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.080536    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9647927d-3657-4748-939a-0014a632ffe5" containerName="main"
Sep 17 18:38:45.085230 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.085213    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"
Sep 17 18:38:45.087970 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.087941    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Sep 17 18:38:45.087970 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.087957    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:38:45.088161 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.087941    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 17 18:38:45.088161 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.088015    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-8x8q7\""
Sep 17 18:38:45.094272 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.094245    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"]
Sep 17 18:38:45.227926 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.225377    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/297b445a-acb9-4967-8993-3b36b5231f48-dshm\") pod \"router-with-refs-test-kserve-5b7c8fb664-4v6k7\" (UID: \"297b445a-acb9-4967-8993-3b36b5231f48\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"
Sep 17 18:38:45.227926 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.225437    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/297b445a-acb9-4967-8993-3b36b5231f48-kserve-provision-location\") pod \"router-with-refs-test-kserve-5b7c8fb664-4v6k7\" (UID: \"297b445a-acb9-4967-8993-3b36b5231f48\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"
Sep 17 18:38:45.227926 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.225501    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/297b445a-acb9-4967-8993-3b36b5231f48-model-cache\") pod \"router-with-refs-test-kserve-5b7c8fb664-4v6k7\" (UID: \"297b445a-acb9-4967-8993-3b36b5231f48\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"
Sep 17 18:38:45.227926 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.225554    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/297b445a-acb9-4967-8993-3b36b5231f48-tls-certs\") pod \"router-with-refs-test-kserve-5b7c8fb664-4v6k7\" (UID: \"297b445a-acb9-4967-8993-3b36b5231f48\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"
Sep 17 18:38:45.227926 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.225586    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnq6t\" (UniqueName: \"kubernetes.io/projected/297b445a-acb9-4967-8993-3b36b5231f48-kube-api-access-xnq6t\") pod \"router-with-refs-test-kserve-5b7c8fb664-4v6k7\" (UID: \"297b445a-acb9-4967-8993-3b36b5231f48\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"
Sep 17 18:38:45.227926 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.225615    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/297b445a-acb9-4967-8993-3b36b5231f48-tmp-dir\") pod \"router-with-refs-test-kserve-5b7c8fb664-4v6k7\" (UID: \"297b445a-acb9-4967-8993-3b36b5231f48\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"
Sep 17 18:38:45.227926 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.225659    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/297b445a-acb9-4967-8993-3b36b5231f48-cabundle-cert\") pod \"router-with-refs-test-kserve-5b7c8fb664-4v6k7\" (UID: \"297b445a-acb9-4967-8993-3b36b5231f48\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"
Sep 17 18:38:45.227926 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.225686    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/297b445a-acb9-4967-8993-3b36b5231f48-home\") pod \"router-with-refs-test-kserve-5b7c8fb664-4v6k7\" (UID: \"297b445a-acb9-4967-8993-3b36b5231f48\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"
Sep 17 18:38:45.326592 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.326536    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/297b445a-acb9-4967-8993-3b36b5231f48-dshm\") pod \"router-with-refs-test-kserve-5b7c8fb664-4v6k7\" (UID: \"297b445a-acb9-4967-8993-3b36b5231f48\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"
Sep 17 18:38:45.326813 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.326605    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/297b445a-acb9-4967-8993-3b36b5231f48-kserve-provision-location\") pod \"router-with-refs-test-kserve-5b7c8fb664-4v6k7\" (UID: \"297b445a-acb9-4967-8993-3b36b5231f48\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"
Sep 17 18:38:45.326813 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.326668    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/297b445a-acb9-4967-8993-3b36b5231f48-model-cache\") pod \"router-with-refs-test-kserve-5b7c8fb664-4v6k7\" (UID: \"297b445a-acb9-4967-8993-3b36b5231f48\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"
Sep 17 18:38:45.326813 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.326724    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/297b445a-acb9-4967-8993-3b36b5231f48-tls-certs\") pod \"router-with-refs-test-kserve-5b7c8fb664-4v6k7\" (UID: \"297b445a-acb9-4967-8993-3b36b5231f48\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"
Sep 17 18:38:45.326813 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.326755    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnq6t\" (UniqueName: \"kubernetes.io/projected/297b445a-acb9-4967-8993-3b36b5231f48-kube-api-access-xnq6t\") pod \"router-with-refs-test-kserve-5b7c8fb664-4v6k7\" (UID: \"297b445a-acb9-4967-8993-3b36b5231f48\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"
Sep 17 18:38:45.326813 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.326786    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/297b445a-acb9-4967-8993-3b36b5231f48-tmp-dir\") pod \"router-with-refs-test-kserve-5b7c8fb664-4v6k7\" (UID: \"297b445a-acb9-4967-8993-3b36b5231f48\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"
Sep 17 18:38:45.327151 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.326827    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/297b445a-acb9-4967-8993-3b36b5231f48-cabundle-cert\") pod \"router-with-refs-test-kserve-5b7c8fb664-4v6k7\" (UID: \"297b445a-acb9-4967-8993-3b36b5231f48\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"
Sep 17 18:38:45.327151 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.326854    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/297b445a-acb9-4967-8993-3b36b5231f48-home\") pod \"router-with-refs-test-kserve-5b7c8fb664-4v6k7\" (UID: \"297b445a-acb9-4967-8993-3b36b5231f48\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"
Sep 17 18:38:45.327355 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.327327    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/297b445a-acb9-4967-8993-3b36b5231f48-home\") pod \"router-with-refs-test-kserve-5b7c8fb664-4v6k7\" (UID: \"297b445a-acb9-4967-8993-3b36b5231f48\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"
Sep 17 18:38:45.328019 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.327990    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/297b445a-acb9-4967-8993-3b36b5231f48-kserve-provision-location\") pod \"router-with-refs-test-kserve-5b7c8fb664-4v6k7\" (UID: \"297b445a-acb9-4967-8993-3b36b5231f48\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"
Sep 17 18:38:45.328143 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.328063    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/297b445a-acb9-4967-8993-3b36b5231f48-tmp-dir\") pod \"router-with-refs-test-kserve-5b7c8fb664-4v6k7\" (UID: \"297b445a-acb9-4967-8993-3b36b5231f48\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"
Sep 17 18:38:45.328491 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.328460    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/297b445a-acb9-4967-8993-3b36b5231f48-cabundle-cert\") pod \"router-with-refs-test-kserve-5b7c8fb664-4v6k7\" (UID: \"297b445a-acb9-4967-8993-3b36b5231f48\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"
Sep 17 18:38:45.328664 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.328638    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/297b445a-acb9-4967-8993-3b36b5231f48-model-cache\") pod \"router-with-refs-test-kserve-5b7c8fb664-4v6k7\" (UID: \"297b445a-acb9-4967-8993-3b36b5231f48\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"
Sep 17 18:38:45.331142 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.331077    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/297b445a-acb9-4967-8993-3b36b5231f48-dshm\") pod \"router-with-refs-test-kserve-5b7c8fb664-4v6k7\" (UID: \"297b445a-acb9-4967-8993-3b36b5231f48\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"
Sep 17 18:38:45.333442 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.333412    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/297b445a-acb9-4967-8993-3b36b5231f48-tls-certs\") pod \"router-with-refs-test-kserve-5b7c8fb664-4v6k7\" (UID: \"297b445a-acb9-4967-8993-3b36b5231f48\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"
Sep 17 18:38:45.336567 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.336515    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnq6t\" (UniqueName: \"kubernetes.io/projected/297b445a-acb9-4967-8993-3b36b5231f48-kube-api-access-xnq6t\") pod \"router-with-refs-test-kserve-5b7c8fb664-4v6k7\" (UID: \"297b445a-acb9-4967-8993-3b36b5231f48\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"
Sep 17 18:38:45.396179 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.396146    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"
Sep 17 18:38:45.546729 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.546697    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4"]
Sep 17 18:38:45.550464 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.550444    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4"
Sep 17 18:38:45.552809 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.552732    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-r8p9l\""
Sep 17 18:38:45.559645 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.559619    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4"]
Sep 17 18:38:45.694445 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.694419    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"]
Sep 17 18:38:45.696531 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:38:45.696504    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod297b445a_acb9_4967_8993_3b36b5231f48.slice/crio-38e4aa0d74dddf33c75b2698b984bdb2cf2b4333f3907de3380e60d0eb19b868 WatchSource:0}: Error finding container 38e4aa0d74dddf33c75b2698b984bdb2cf2b4333f3907de3380e60d0eb19b868: Status 404 returned error can't find the container with id 38e4aa0d74dddf33c75b2698b984bdb2cf2b4333f3907de3380e60d0eb19b868
Sep 17 18:38:45.732179 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.732139    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsxb9\" (UniqueName: \"kubernetes.io/projected/30c76801-5c82-46c7-9b6b-23309acd3bb8-kube-api-access-xsxb9\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4\" (UID: \"30c76801-5c82-46c7-9b6b-23309acd3bb8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4"
Sep 17 18:38:45.732379 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.732196    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30c76801-5c82-46c7-9b6b-23309acd3bb8-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4\" (UID: \"30c76801-5c82-46c7-9b6b-23309acd3bb8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4"
Sep 17 18:38:45.833035 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.832995    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsxb9\" (UniqueName: \"kubernetes.io/projected/30c76801-5c82-46c7-9b6b-23309acd3bb8-kube-api-access-xsxb9\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4\" (UID: \"30c76801-5c82-46c7-9b6b-23309acd3bb8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4"
Sep 17 18:38:45.833254 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.833059    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30c76801-5c82-46c7-9b6b-23309acd3bb8-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4\" (UID: \"30c76801-5c82-46c7-9b6b-23309acd3bb8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4"
Sep 17 18:38:45.835791 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.835758    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30c76801-5c82-46c7-9b6b-23309acd3bb8-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4\" (UID: \"30c76801-5c82-46c7-9b6b-23309acd3bb8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4"
Sep 17 18:38:45.841947 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.841920    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsxb9\" (UniqueName: \"kubernetes.io/projected/30c76801-5c82-46c7-9b6b-23309acd3bb8-kube-api-access-xsxb9\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4\" (UID: \"30c76801-5c82-46c7-9b6b-23309acd3bb8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4"
Sep 17 18:38:45.861991 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:45.861957    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4"
Sep 17 18:38:46.157637 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:46.157607    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4"]
Sep 17 18:38:46.159862 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:38:46.159831    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c76801_5c82_46c7_9b6b_23309acd3bb8.slice/crio-f3417bc11ba02726ed140c8559bf9471d0427403f4184a8b795890eb09c44e41 WatchSource:0}: Error finding container f3417bc11ba02726ed140c8559bf9471d0427403f4184a8b795890eb09c44e41: Status 404 returned error can't find the container with id f3417bc11ba02726ed140c8559bf9471d0427403f4184a8b795890eb09c44e41
Sep 17 18:38:46.388623 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:46.388586    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4" event={"ID":"30c76801-5c82-46c7-9b6b-23309acd3bb8","Type":"ContainerStarted","Data":"736ddc52fc1e76d8f55fe866d788347be6bfc30a839c24450251528182a5ca40"}
Sep 17 18:38:46.388813 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:46.388654    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4" event={"ID":"30c76801-5c82-46c7-9b6b-23309acd3bb8","Type":"ContainerStarted","Data":"f3417bc11ba02726ed140c8559bf9471d0427403f4184a8b795890eb09c44e41"}
Sep 17 18:38:46.388813 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:46.388698    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4"
Sep 17 18:38:46.390280 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:46.390236    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7" event={"ID":"297b445a-acb9-4967-8993-3b36b5231f48","Type":"ContainerStarted","Data":"db6e26b4b44612b59502ce469ba4caa7c85102e99a09f452430d7b1d28b3088f"}
Sep 17 18:38:46.390440 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:46.390298    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7" event={"ID":"297b445a-acb9-4967-8993-3b36b5231f48","Type":"ContainerStarted","Data":"38e4aa0d74dddf33c75b2698b984bdb2cf2b4333f3907de3380e60d0eb19b868"}
Sep 17 18:38:46.405452 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:46.405392    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4" podStartSLOduration=1.4053714990000001 podStartE2EDuration="1.405371499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:38:46.403688191 +0000 UTC m=+2514.494309842" watchObservedRunningTime="2026-09-17 18:38:46.405371499 +0000 UTC m=+2514.495993150"
Sep 17 18:38:48.403953 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:48.403879    2581 generic.go:356] "Generic (PLEG): container finished" podID="297b445a-acb9-4967-8993-3b36b5231f48" containerID="db6e26b4b44612b59502ce469ba4caa7c85102e99a09f452430d7b1d28b3088f" exitCode=0
Sep 17 18:38:48.404446 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:48.403971    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7" event={"ID":"297b445a-acb9-4967-8993-3b36b5231f48","Type":"ContainerDied","Data":"db6e26b4b44612b59502ce469ba4caa7c85102e99a09f452430d7b1d28b3088f"}
Sep 17 18:38:49.412128 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:49.412090    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7" event={"ID":"297b445a-acb9-4967-8993-3b36b5231f48","Type":"ContainerStarted","Data":"bf29f64464ddeff3430de1712d9b0f6180a155548a46d66dca618ce37a569ffb"}
Sep 17 18:38:49.434485 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:49.434420    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7" podStartSLOduration=1.8899753860000001 podStartE2EDuration="4.434400026s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.54442464s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:38:49.433152717 +0000 UTC m=+2517.523774370" watchObservedRunningTime="2026-09-17 18:38:49.434400026 +0000 UTC m=+2517.525021677"
Sep 17 18:38:50.396665 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:50.396623    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"
Sep 17 18:38:50.419672 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:50.419640    2581 generic.go:356] "Generic (PLEG): container finished" podID="47140786-7812-4af5-85c6-5528ce2bdd3a" containerID="6d6782d230ec4bbfea00e673ee075729176d147fc746217d52abb8c615fa817e" exitCode=0
Sep 17 18:38:50.420061 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:50.419711    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx" event={"ID":"47140786-7812-4af5-85c6-5528ce2bdd3a","Type":"ContainerDied","Data":"6d6782d230ec4bbfea00e673ee075729176d147fc746217d52abb8c615fa817e"}
Sep 17 18:38:50.420217 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:50.420203    2581 scope.go:117] "RemoveContainer" containerID="6d6782d230ec4bbfea00e673ee075729176d147fc746217d52abb8c615fa817e"
Sep 17 18:38:50.421186 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:50.421163    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:38:50.421249 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:50.421224    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:38:50.421285 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:50.421256    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:38:51.430983 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:51.430938    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx" event={"ID":"47140786-7812-4af5-85c6-5528ce2bdd3a","Type":"ContainerStarted","Data":"767ae479458db4ede472c30d9657fa12063c94492326511272846a607806b0a2"}
Sep 17 18:38:51.463465 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:51.463429    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:51.463685 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:51.463653    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx" podUID="47140786-7812-4af5-85c6-5528ce2bdd3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 17 18:38:52.463162 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:52.463108    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx" podUID="47140786-7812-4af5-85c6-5528ce2bdd3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 17 18:38:53.463391 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:53.463350    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx" podUID="47140786-7812-4af5-85c6-5528ce2bdd3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 17 18:38:54.463705 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:54.463663    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx" podUID="47140786-7812-4af5-85c6-5528ce2bdd3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 17 18:38:55.396754 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:55.396706    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"
Sep 17 18:38:55.398435 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:55.398402    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7" podUID="297b445a-acb9-4967-8993-3b36b5231f48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 18:38:55.463859 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:55.463818    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx" podUID="47140786-7812-4af5-85c6-5528ce2bdd3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 17 18:38:56.361886 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:56.361854    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:56.463751 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:56.463705    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx" podUID="47140786-7812-4af5-85c6-5528ce2bdd3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 17 18:38:57.467549 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:57.467515    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:38:57.468571 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:38:57.468549    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:39:05.397237 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:39:05.397189    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7" podUID="297b445a-acb9-4967-8993-3b36b5231f48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 18:39:15.396986 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:39:15.396931    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7" podUID="297b445a-acb9-4967-8993-3b36b5231f48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 18:39:17.397567 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:39:17.397537    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4"
Sep 17 18:39:25.397262 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:39:25.397216    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7" podUID="297b445a-acb9-4967-8993-3b36b5231f48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 18:39:35.397409 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:39:35.397362    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7" podUID="297b445a-acb9-4967-8993-3b36b5231f48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 18:39:45.396797 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:39:45.396753    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7" podUID="297b445a-acb9-4967-8993-3b36b5231f48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 18:39:55.397333 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:39:55.397282    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7" podUID="297b445a-acb9-4967-8993-3b36b5231f48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 18:40:05.397106 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:40:05.397057    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7" podUID="297b445a-acb9-4967-8993-3b36b5231f48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 18:40:15.396767 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:40:15.396656    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7" podUID="297b445a-acb9-4967-8993-3b36b5231f48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 18:40:25.397502 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:40:25.397451    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7" podUID="297b445a-acb9-4967-8993-3b36b5231f48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 18:40:35.397343 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:40:35.397281    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7" podUID="297b445a-acb9-4967-8993-3b36b5231f48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 18:40:45.406743 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:40:45.406709    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"
Sep 17 18:40:45.414483 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:40:45.414454    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"
Sep 17 18:40:49.897270 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:40:49.897232    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"]
Sep 17 18:40:49.899189 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:40:49.899163    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4"]
Sep 17 18:40:57.397344 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:40:57.397306    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4" podUID="30c76801-5c82-46c7-9b6b-23309acd3bb8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:41:04.918797 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:04.918733    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4" podUID="30c76801-5c82-46c7-9b6b-23309acd3bb8" containerName="main" containerID="cri-o://736ddc52fc1e76d8f55fe866d788347be6bfc30a839c24450251528182a5ca40" gracePeriod=45
Sep 17 18:41:04.920534 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:04.920490    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7" podUID="297b445a-acb9-4967-8993-3b36b5231f48" containerName="main" containerID="cri-o://bf29f64464ddeff3430de1712d9b0f6180a155548a46d66dca618ce37a569ffb" gracePeriod=45
Sep 17 18:41:05.407503 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:05.407459    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7" podUID="297b445a-acb9-4967-8993-3b36b5231f48" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 17 18:41:05.672318 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:05.672256    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"
Sep 17 18:41:05.743166 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:05.743127    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/297b445a-acb9-4967-8993-3b36b5231f48-kserve-provision-location\") pod \"297b445a-acb9-4967-8993-3b36b5231f48\" (UID: \"297b445a-acb9-4967-8993-3b36b5231f48\") "
Sep 17 18:41:05.743373 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:05.743174    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnq6t\" (UniqueName: \"kubernetes.io/projected/297b445a-acb9-4967-8993-3b36b5231f48-kube-api-access-xnq6t\") pod \"297b445a-acb9-4967-8993-3b36b5231f48\" (UID: \"297b445a-acb9-4967-8993-3b36b5231f48\") "
Sep 17 18:41:05.743373 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:05.743220    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/297b445a-acb9-4967-8993-3b36b5231f48-tmp-dir\") pod \"297b445a-acb9-4967-8993-3b36b5231f48\" (UID: \"297b445a-acb9-4967-8993-3b36b5231f48\") "
Sep 17 18:41:05.743373 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:05.743247    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/297b445a-acb9-4967-8993-3b36b5231f48-cabundle-cert\") pod \"297b445a-acb9-4967-8993-3b36b5231f48\" (UID: \"297b445a-acb9-4967-8993-3b36b5231f48\") "
Sep 17 18:41:05.743373 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:05.743313    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/297b445a-acb9-4967-8993-3b36b5231f48-model-cache\") pod \"297b445a-acb9-4967-8993-3b36b5231f48\" (UID: \"297b445a-acb9-4967-8993-3b36b5231f48\") "
Sep 17 18:41:05.743373 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:05.743341    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/297b445a-acb9-4967-8993-3b36b5231f48-tls-certs\") pod \"297b445a-acb9-4967-8993-3b36b5231f48\" (UID: \"297b445a-acb9-4967-8993-3b36b5231f48\") "
Sep 17 18:41:05.743373 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:05.743366    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/297b445a-acb9-4967-8993-3b36b5231f48-home\") pod \"297b445a-acb9-4967-8993-3b36b5231f48\" (UID: \"297b445a-acb9-4967-8993-3b36b5231f48\") "
Sep 17 18:41:05.743688 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:05.743529    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/297b445a-acb9-4967-8993-3b36b5231f48-dshm\") pod \"297b445a-acb9-4967-8993-3b36b5231f48\" (UID: \"297b445a-acb9-4967-8993-3b36b5231f48\") "
Sep 17 18:41:05.743688 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:05.743633    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297b445a-acb9-4967-8993-3b36b5231f48-model-cache" (OuterVolumeSpecName: "model-cache") pod "297b445a-acb9-4967-8993-3b36b5231f48" (UID: "297b445a-acb9-4967-8993-3b36b5231f48"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:41:05.743895 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:05.743848    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/297b445a-acb9-4967-8993-3b36b5231f48-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "297b445a-acb9-4967-8993-3b36b5231f48" (UID: "297b445a-acb9-4967-8993-3b36b5231f48"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:05.744148 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:05.744114    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/297b445a-acb9-4967-8993-3b36b5231f48-cabundle-cert\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:41:05.744148 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:05.744140    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/297b445a-acb9-4967-8993-3b36b5231f48-model-cache\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:41:05.744278 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:05.744198    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297b445a-acb9-4967-8993-3b36b5231f48-home" (OuterVolumeSpecName: "home") pod "297b445a-acb9-4967-8993-3b36b5231f48" (UID: "297b445a-acb9-4967-8993-3b36b5231f48"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:41:05.745766 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:05.745733    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/297b445a-acb9-4967-8993-3b36b5231f48-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "297b445a-acb9-4967-8993-3b36b5231f48" (UID: "297b445a-acb9-4967-8993-3b36b5231f48"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:41:05.745857 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:05.745832    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/297b445a-acb9-4967-8993-3b36b5231f48-kube-api-access-xnq6t" (OuterVolumeSpecName: "kube-api-access-xnq6t") pod "297b445a-acb9-4967-8993-3b36b5231f48" (UID: "297b445a-acb9-4967-8993-3b36b5231f48"). InnerVolumeSpecName "kube-api-access-xnq6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:41:05.746069 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:05.746047    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297b445a-acb9-4967-8993-3b36b5231f48-dshm" (OuterVolumeSpecName: "dshm") pod "297b445a-acb9-4967-8993-3b36b5231f48" (UID: "297b445a-acb9-4967-8993-3b36b5231f48"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:41:05.756464 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:05.756424    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297b445a-acb9-4967-8993-3b36b5231f48-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "297b445a-acb9-4967-8993-3b36b5231f48" (UID: "297b445a-acb9-4967-8993-3b36b5231f48"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:41:05.764385 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:05.764342    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297b445a-acb9-4967-8993-3b36b5231f48-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "297b445a-acb9-4967-8993-3b36b5231f48" (UID: "297b445a-acb9-4967-8993-3b36b5231f48"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:41:05.844956 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:05.844897    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/297b445a-acb9-4967-8993-3b36b5231f48-dshm\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:41:05.844956 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:05.844955    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/297b445a-acb9-4967-8993-3b36b5231f48-kserve-provision-location\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:41:05.845160 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:05.844965    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnq6t\" (UniqueName: \"kubernetes.io/projected/297b445a-acb9-4967-8993-3b36b5231f48-kube-api-access-xnq6t\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:41:05.845160 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:05.844975    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/297b445a-acb9-4967-8993-3b36b5231f48-tmp-dir\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:41:05.845160 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:05.844983    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/297b445a-acb9-4967-8993-3b36b5231f48-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:41:05.845160 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:05.844991    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/297b445a-acb9-4967-8993-3b36b5231f48-home\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:41:06.121398 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:06.121363    2581 generic.go:356] "Generic (PLEG): container finished" podID="297b445a-acb9-4967-8993-3b36b5231f48" containerID="bf29f64464ddeff3430de1712d9b0f6180a155548a46d66dca618ce37a569ffb" exitCode=0
Sep 17 18:41:06.121802 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:06.121424    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"
Sep 17 18:41:06.121802 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:06.121447    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7" event={"ID":"297b445a-acb9-4967-8993-3b36b5231f48","Type":"ContainerDied","Data":"bf29f64464ddeff3430de1712d9b0f6180a155548a46d66dca618ce37a569ffb"}
Sep 17 18:41:06.121802 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:06.121506    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7" event={"ID":"297b445a-acb9-4967-8993-3b36b5231f48","Type":"ContainerDied","Data":"38e4aa0d74dddf33c75b2698b984bdb2cf2b4333f3907de3380e60d0eb19b868"}
Sep 17 18:41:06.121802 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:06.121529    2581 scope.go:117] "RemoveContainer" containerID="bf29f64464ddeff3430de1712d9b0f6180a155548a46d66dca618ce37a569ffb"
Sep 17 18:41:06.130746 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:06.130729    2581 scope.go:117] "RemoveContainer" containerID="db6e26b4b44612b59502ce469ba4caa7c85102e99a09f452430d7b1d28b3088f"
Sep 17 18:41:06.138752 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:06.138732    2581 scope.go:117] "RemoveContainer" containerID="bf29f64464ddeff3430de1712d9b0f6180a155548a46d66dca618ce37a569ffb"
Sep 17 18:41:06.139028 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:41:06.139011    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf29f64464ddeff3430de1712d9b0f6180a155548a46d66dca618ce37a569ffb\": container with ID starting with bf29f64464ddeff3430de1712d9b0f6180a155548a46d66dca618ce37a569ffb not found: ID does not exist" containerID="bf29f64464ddeff3430de1712d9b0f6180a155548a46d66dca618ce37a569ffb"
Sep 17 18:41:06.139083 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:06.139036    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf29f64464ddeff3430de1712d9b0f6180a155548a46d66dca618ce37a569ffb"} err="failed to get container status \"bf29f64464ddeff3430de1712d9b0f6180a155548a46d66dca618ce37a569ffb\": rpc error: code = NotFound desc = could not find container \"bf29f64464ddeff3430de1712d9b0f6180a155548a46d66dca618ce37a569ffb\": container with ID starting with bf29f64464ddeff3430de1712d9b0f6180a155548a46d66dca618ce37a569ffb not found: ID does not exist"
Sep 17 18:41:06.139083 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:06.139051    2581 scope.go:117] "RemoveContainer" containerID="db6e26b4b44612b59502ce469ba4caa7c85102e99a09f452430d7b1d28b3088f"
Sep 17 18:41:06.139273 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:41:06.139257    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db6e26b4b44612b59502ce469ba4caa7c85102e99a09f452430d7b1d28b3088f\": container with ID starting with db6e26b4b44612b59502ce469ba4caa7c85102e99a09f452430d7b1d28b3088f not found: ID does not exist" containerID="db6e26b4b44612b59502ce469ba4caa7c85102e99a09f452430d7b1d28b3088f"
Sep 17 18:41:06.139314 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:06.139275    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db6e26b4b44612b59502ce469ba4caa7c85102e99a09f452430d7b1d28b3088f"} err="failed to get container status \"db6e26b4b44612b59502ce469ba4caa7c85102e99a09f452430d7b1d28b3088f\": rpc error: code = NotFound desc = could not find container \"db6e26b4b44612b59502ce469ba4caa7c85102e99a09f452430d7b1d28b3088f\": container with ID starting with db6e26b4b44612b59502ce469ba4caa7c85102e99a09f452430d7b1d28b3088f not found: ID does not exist"
Sep 17 18:41:06.144411 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:06.144384    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"]
Sep 17 18:41:06.149708 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:06.149689    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5b7c8fb664-4v6k7"]
Sep 17 18:41:06.357934 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:06.357881    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="297b445a-acb9-4967-8993-3b36b5231f48" path="/var/lib/kubelet/pods/297b445a-acb9-4967-8993-3b36b5231f48/volumes"
Sep 17 18:41:07.397546 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:07.397507    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4" podUID="30c76801-5c82-46c7-9b6b-23309acd3bb8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:41:17.397951 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:17.397880    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4" podUID="30c76801-5c82-46c7-9b6b-23309acd3bb8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:41:17.398371 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:17.398018    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4"
Sep 17 18:41:27.398027 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:27.397984    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4" podUID="30c76801-5c82-46c7-9b6b-23309acd3bb8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:41:35.171389 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:35.171360    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4"
Sep 17 18:41:35.255317 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:35.255272    2581 generic.go:356] "Generic (PLEG): container finished" podID="30c76801-5c82-46c7-9b6b-23309acd3bb8" containerID="736ddc52fc1e76d8f55fe866d788347be6bfc30a839c24450251528182a5ca40" exitCode=0
Sep 17 18:41:35.255317 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:35.255321    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4"
Sep 17 18:41:35.255531 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:35.255353    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4" event={"ID":"30c76801-5c82-46c7-9b6b-23309acd3bb8","Type":"ContainerDied","Data":"736ddc52fc1e76d8f55fe866d788347be6bfc30a839c24450251528182a5ca40"}
Sep 17 18:41:35.255531 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:35.255411    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4" event={"ID":"30c76801-5c82-46c7-9b6b-23309acd3bb8","Type":"ContainerDied","Data":"f3417bc11ba02726ed140c8559bf9471d0427403f4184a8b795890eb09c44e41"}
Sep 17 18:41:35.255531 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:35.255435    2581 scope.go:117] "RemoveContainer" containerID="736ddc52fc1e76d8f55fe866d788347be6bfc30a839c24450251528182a5ca40"
Sep 17 18:41:35.263817 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:35.263796    2581 scope.go:117] "RemoveContainer" containerID="736ddc52fc1e76d8f55fe866d788347be6bfc30a839c24450251528182a5ca40"
Sep 17 18:41:35.264127 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:41:35.264106    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"736ddc52fc1e76d8f55fe866d788347be6bfc30a839c24450251528182a5ca40\": container with ID starting with 736ddc52fc1e76d8f55fe866d788347be6bfc30a839c24450251528182a5ca40 not found: ID does not exist" containerID="736ddc52fc1e76d8f55fe866d788347be6bfc30a839c24450251528182a5ca40"
Sep 17 18:41:35.264187 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:35.264139    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"736ddc52fc1e76d8f55fe866d788347be6bfc30a839c24450251528182a5ca40"} err="failed to get container status \"736ddc52fc1e76d8f55fe866d788347be6bfc30a839c24450251528182a5ca40\": rpc error: code = NotFound desc = could not find container \"736ddc52fc1e76d8f55fe866d788347be6bfc30a839c24450251528182a5ca40\": container with ID starting with 736ddc52fc1e76d8f55fe866d788347be6bfc30a839c24450251528182a5ca40 not found: ID does not exist"
Sep 17 18:41:35.318573 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:35.318541    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30c76801-5c82-46c7-9b6b-23309acd3bb8-tls-certs\") pod \"30c76801-5c82-46c7-9b6b-23309acd3bb8\" (UID: \"30c76801-5c82-46c7-9b6b-23309acd3bb8\") "
Sep 17 18:41:35.318743 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:35.318606    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsxb9\" (UniqueName: \"kubernetes.io/projected/30c76801-5c82-46c7-9b6b-23309acd3bb8-kube-api-access-xsxb9\") pod \"30c76801-5c82-46c7-9b6b-23309acd3bb8\" (UID: \"30c76801-5c82-46c7-9b6b-23309acd3bb8\") "
Sep 17 18:41:35.320779 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:35.320751    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30c76801-5c82-46c7-9b6b-23309acd3bb8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "30c76801-5c82-46c7-9b6b-23309acd3bb8" (UID: "30c76801-5c82-46c7-9b6b-23309acd3bb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:41:35.320877 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:35.320818    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c76801-5c82-46c7-9b6b-23309acd3bb8-kube-api-access-xsxb9" (OuterVolumeSpecName: "kube-api-access-xsxb9") pod "30c76801-5c82-46c7-9b6b-23309acd3bb8" (UID: "30c76801-5c82-46c7-9b6b-23309acd3bb8"). InnerVolumeSpecName "kube-api-access-xsxb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:41:35.419835 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:35.419725    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30c76801-5c82-46c7-9b6b-23309acd3bb8-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:41:35.419835 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:35.419768    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsxb9\" (UniqueName: \"kubernetes.io/projected/30c76801-5c82-46c7-9b6b-23309acd3bb8-kube-api-access-xsxb9\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:41:35.577401 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:35.577365    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4"]
Sep 17 18:41:35.580817 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:35.580792    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-5vkr4"]
Sep 17 18:41:36.359008 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:36.358969    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30c76801-5c82-46c7-9b6b-23309acd3bb8" path="/var/lib/kubelet/pods/30c76801-5c82-46c7-9b6b-23309acd3bb8/volumes"
Sep 17 18:41:44.869535 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:44.869492    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"]
Sep 17 18:41:44.870057 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:44.869804    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx" podUID="47140786-7812-4af5-85c6-5528ce2bdd3a" containerName="istio-proxy" containerID="cri-o://767ae479458db4ede472c30d9657fa12063c94492326511272846a607806b0a2" gracePeriod=30
Sep 17 18:41:45.360432 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.360394    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"]
Sep 17 18:41:45.360993 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.360962    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30c76801-5c82-46c7-9b6b-23309acd3bb8" containerName="main"
Sep 17 18:41:45.360993 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.360988    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="30c76801-5c82-46c7-9b6b-23309acd3bb8" containerName="main"
Sep 17 18:41:45.361153 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.361030    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="297b445a-acb9-4967-8993-3b36b5231f48" containerName="storage-initializer"
Sep 17 18:41:45.361153 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.361039    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="297b445a-acb9-4967-8993-3b36b5231f48" containerName="storage-initializer"
Sep 17 18:41:45.361153 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.361049    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="297b445a-acb9-4967-8993-3b36b5231f48" containerName="main"
Sep 17 18:41:45.361153 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.361057    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="297b445a-acb9-4967-8993-3b36b5231f48" containerName="main"
Sep 17 18:41:45.361153 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.361140    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="297b445a-acb9-4967-8993-3b36b5231f48" containerName="main"
Sep 17 18:41:45.361325 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.361158    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="30c76801-5c82-46c7-9b6b-23309acd3bb8" containerName="main"
Sep 17 18:41:45.366145 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.366122    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"
Sep 17 18:41:45.368623 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.368590    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:41:45.369026 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.369006    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 17 18:41:45.369316 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.369299    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-p7k6s\""
Sep 17 18:41:45.369400 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.369320    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 17 18:41:45.369461 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.369395    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 17 18:41:45.369461 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.369425    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 17 18:41:45.375163 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.375139    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"]
Sep 17 18:41:45.509965 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.509927    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b632c1dd-b97e-4856-ac3f-96d38201688b-home\") pod \"stop-feature-test-kserve-66fd6b68fd-h5dlj\" (UID: \"b632c1dd-b97e-4856-ac3f-96d38201688b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"
Sep 17 18:41:45.510174 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.509988    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98h7q\" (UniqueName: \"kubernetes.io/projected/b632c1dd-b97e-4856-ac3f-96d38201688b-kube-api-access-98h7q\") pod \"stop-feature-test-kserve-66fd6b68fd-h5dlj\" (UID: \"b632c1dd-b97e-4856-ac3f-96d38201688b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"
Sep 17 18:41:45.510174 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.510010    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b632c1dd-b97e-4856-ac3f-96d38201688b-tmp-dir\") pod \"stop-feature-test-kserve-66fd6b68fd-h5dlj\" (UID: \"b632c1dd-b97e-4856-ac3f-96d38201688b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"
Sep 17 18:41:45.510310 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.510207    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b632c1dd-b97e-4856-ac3f-96d38201688b-dshm\") pod \"stop-feature-test-kserve-66fd6b68fd-h5dlj\" (UID: \"b632c1dd-b97e-4856-ac3f-96d38201688b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"
Sep 17 18:41:45.510310 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.510292    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b632c1dd-b97e-4856-ac3f-96d38201688b-tls-certs\") pod \"stop-feature-test-kserve-66fd6b68fd-h5dlj\" (UID: \"b632c1dd-b97e-4856-ac3f-96d38201688b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"
Sep 17 18:41:45.510427 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.510348    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b632c1dd-b97e-4856-ac3f-96d38201688b-model-cache\") pod \"stop-feature-test-kserve-66fd6b68fd-h5dlj\" (UID: \"b632c1dd-b97e-4856-ac3f-96d38201688b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"
Sep 17 18:41:45.510570 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.510546    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b632c1dd-b97e-4856-ac3f-96d38201688b-kserve-provision-location\") pod \"stop-feature-test-kserve-66fd6b68fd-h5dlj\" (UID: \"b632c1dd-b97e-4856-ac3f-96d38201688b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"
Sep 17 18:41:45.510705 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.510642    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b632c1dd-b97e-4856-ac3f-96d38201688b-cabundle-cert\") pod \"stop-feature-test-kserve-66fd6b68fd-h5dlj\" (UID: \"b632c1dd-b97e-4856-ac3f-96d38201688b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"
Sep 17 18:41:45.611639 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.611547    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b632c1dd-b97e-4856-ac3f-96d38201688b-kserve-provision-location\") pod \"stop-feature-test-kserve-66fd6b68fd-h5dlj\" (UID: \"b632c1dd-b97e-4856-ac3f-96d38201688b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"
Sep 17 18:41:45.611639 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.611591    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b632c1dd-b97e-4856-ac3f-96d38201688b-cabundle-cert\") pod \"stop-feature-test-kserve-66fd6b68fd-h5dlj\" (UID: \"b632c1dd-b97e-4856-ac3f-96d38201688b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"
Sep 17 18:41:45.611877 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.611689    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b632c1dd-b97e-4856-ac3f-96d38201688b-home\") pod \"stop-feature-test-kserve-66fd6b68fd-h5dlj\" (UID: \"b632c1dd-b97e-4856-ac3f-96d38201688b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"
Sep 17 18:41:45.611877 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.611713    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98h7q\" (UniqueName: \"kubernetes.io/projected/b632c1dd-b97e-4856-ac3f-96d38201688b-kube-api-access-98h7q\") pod \"stop-feature-test-kserve-66fd6b68fd-h5dlj\" (UID: \"b632c1dd-b97e-4856-ac3f-96d38201688b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"
Sep 17 18:41:45.611877 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.611731    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b632c1dd-b97e-4856-ac3f-96d38201688b-tmp-dir\") pod \"stop-feature-test-kserve-66fd6b68fd-h5dlj\" (UID: \"b632c1dd-b97e-4856-ac3f-96d38201688b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"
Sep 17 18:41:45.611877 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.611765    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b632c1dd-b97e-4856-ac3f-96d38201688b-dshm\") pod \"stop-feature-test-kserve-66fd6b68fd-h5dlj\" (UID: \"b632c1dd-b97e-4856-ac3f-96d38201688b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"
Sep 17 18:41:45.611877 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.611815    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b632c1dd-b97e-4856-ac3f-96d38201688b-tls-certs\") pod \"stop-feature-test-kserve-66fd6b68fd-h5dlj\" (UID: \"b632c1dd-b97e-4856-ac3f-96d38201688b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"
Sep 17 18:41:45.611877 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.611854    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b632c1dd-b97e-4856-ac3f-96d38201688b-model-cache\") pod \"stop-feature-test-kserve-66fd6b68fd-h5dlj\" (UID: \"b632c1dd-b97e-4856-ac3f-96d38201688b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"
Sep 17 18:41:45.612194 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.612092    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b632c1dd-b97e-4856-ac3f-96d38201688b-kserve-provision-location\") pod \"stop-feature-test-kserve-66fd6b68fd-h5dlj\" (UID: \"b632c1dd-b97e-4856-ac3f-96d38201688b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"
Sep 17 18:41:45.612368 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.612342    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b632c1dd-b97e-4856-ac3f-96d38201688b-tmp-dir\") pod \"stop-feature-test-kserve-66fd6b68fd-h5dlj\" (UID: \"b632c1dd-b97e-4856-ac3f-96d38201688b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"
Sep 17 18:41:45.612441 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.612346    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b632c1dd-b97e-4856-ac3f-96d38201688b-model-cache\") pod \"stop-feature-test-kserve-66fd6b68fd-h5dlj\" (UID: \"b632c1dd-b97e-4856-ac3f-96d38201688b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"
Sep 17 18:41:45.612742 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.612717    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b632c1dd-b97e-4856-ac3f-96d38201688b-home\") pod \"stop-feature-test-kserve-66fd6b68fd-h5dlj\" (UID: \"b632c1dd-b97e-4856-ac3f-96d38201688b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"
Sep 17 18:41:45.613159 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.613140    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b632c1dd-b97e-4856-ac3f-96d38201688b-cabundle-cert\") pod \"stop-feature-test-kserve-66fd6b68fd-h5dlj\" (UID: \"b632c1dd-b97e-4856-ac3f-96d38201688b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"
Sep 17 18:41:45.614868 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.614841    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b632c1dd-b97e-4856-ac3f-96d38201688b-dshm\") pod \"stop-feature-test-kserve-66fd6b68fd-h5dlj\" (UID: \"b632c1dd-b97e-4856-ac3f-96d38201688b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"
Sep 17 18:41:45.615014 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.614992    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b632c1dd-b97e-4856-ac3f-96d38201688b-tls-certs\") pod \"stop-feature-test-kserve-66fd6b68fd-h5dlj\" (UID: \"b632c1dd-b97e-4856-ac3f-96d38201688b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"
Sep 17 18:41:45.620609 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.620580    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l"]
Sep 17 18:41:45.623049 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.623024    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98h7q\" (UniqueName: \"kubernetes.io/projected/b632c1dd-b97e-4856-ac3f-96d38201688b-kube-api-access-98h7q\") pod \"stop-feature-test-kserve-66fd6b68fd-h5dlj\" (UID: \"b632c1dd-b97e-4856-ac3f-96d38201688b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"
Sep 17 18:41:45.625829 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.625808    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l"
Sep 17 18:41:45.628330 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.628308    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-rm5xd\""
Sep 17 18:41:45.638008 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.637972    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l"]
Sep 17 18:41:45.679000 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.678956    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"
Sep 17 18:41:45.813161 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.813120    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccxsx\" (UniqueName: \"kubernetes.io/projected/612a768e-20be-4759-a1b9-15611fd9d28e-kube-api-access-ccxsx\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l\" (UID: \"612a768e-20be-4759-a1b9-15611fd9d28e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l"
Sep 17 18:41:45.813161 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.813169    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/612a768e-20be-4759-a1b9-15611fd9d28e-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l\" (UID: \"612a768e-20be-4759-a1b9-15611fd9d28e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l"
Sep 17 18:41:45.914663 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.914555    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccxsx\" (UniqueName: \"kubernetes.io/projected/612a768e-20be-4759-a1b9-15611fd9d28e-kube-api-access-ccxsx\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l\" (UID: \"612a768e-20be-4759-a1b9-15611fd9d28e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l"
Sep 17 18:41:45.914663 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.914622    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/612a768e-20be-4759-a1b9-15611fd9d28e-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l\" (UID: \"612a768e-20be-4759-a1b9-15611fd9d28e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l"
Sep 17 18:41:45.917150 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.917129    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/612a768e-20be-4759-a1b9-15611fd9d28e-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l\" (UID: \"612a768e-20be-4759-a1b9-15611fd9d28e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l"
Sep 17 18:41:45.924370 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.924329    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccxsx\" (UniqueName: \"kubernetes.io/projected/612a768e-20be-4759-a1b9-15611fd9d28e-kube-api-access-ccxsx\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l\" (UID: \"612a768e-20be-4759-a1b9-15611fd9d28e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l"
Sep 17 18:41:45.949580 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.949540    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l"
Sep 17 18:41:45.972129 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:45.972099    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"]
Sep 17 18:41:45.975401 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:41:45.975373    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb632c1dd_b97e_4856_ac3f_96d38201688b.slice/crio-38e53794f8c7ee5920df16c4188f07d2652b9e8d241c49172b296e36592ce909 WatchSource:0}: Error finding container 38e53794f8c7ee5920df16c4188f07d2652b9e8d241c49172b296e36592ce909: Status 404 returned error can't find the container with id 38e53794f8c7ee5920df16c4188f07d2652b9e8d241c49172b296e36592ce909
Sep 17 18:41:46.225338 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:46.225287    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l"]
Sep 17 18:41:46.227996 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:41:46.227962    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod612a768e_20be_4759_a1b9_15611fd9d28e.slice/crio-b33789d9c1f9e5fcac62122090a15ff549630782b74fab99021d06f706c176fd WatchSource:0}: Error finding container b33789d9c1f9e5fcac62122090a15ff549630782b74fab99021d06f706c176fd: Status 404 returned error can't find the container with id b33789d9c1f9e5fcac62122090a15ff549630782b74fab99021d06f706c176fd
Sep 17 18:41:46.310765 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:46.310726    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj" event={"ID":"b632c1dd-b97e-4856-ac3f-96d38201688b","Type":"ContainerStarted","Data":"76d34bb084e05745666ad9119dc63f53102b471918115e4cb6a4ba8964546fd8"}
Sep 17 18:41:46.310974 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:46.310800    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj" event={"ID":"b632c1dd-b97e-4856-ac3f-96d38201688b","Type":"ContainerStarted","Data":"38e53794f8c7ee5920df16c4188f07d2652b9e8d241c49172b296e36592ce909"}
Sep 17 18:41:46.312516 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:46.312483    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l" event={"ID":"612a768e-20be-4759-a1b9-15611fd9d28e","Type":"ContainerStarted","Data":"05c2d8de8182c8b07df5317d933fe5fdbdcecf9a73d0bcc5facd7aa1a295cf76"}
Sep 17 18:41:46.312641 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:46.312542    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l" event={"ID":"612a768e-20be-4759-a1b9-15611fd9d28e","Type":"ContainerStarted","Data":"b33789d9c1f9e5fcac62122090a15ff549630782b74fab99021d06f706c176fd"}
Sep 17 18:41:46.312641 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:46.312573    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l"
Sep 17 18:41:46.347354 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:46.347295    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l" podStartSLOduration=1.34727514 podStartE2EDuration="1.34727514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:41:46.345542189 +0000 UTC m=+2694.436163840" watchObservedRunningTime="2026-09-17 18:41:46.34727514 +0000 UTC m=+2694.437896790"
Sep 17 18:41:49.330002 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:49.329968    2581 generic.go:356] "Generic (PLEG): container finished" podID="b632c1dd-b97e-4856-ac3f-96d38201688b" containerID="76d34bb084e05745666ad9119dc63f53102b471918115e4cb6a4ba8964546fd8" exitCode=0
Sep 17 18:41:49.330379 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:49.330040    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj" event={"ID":"b632c1dd-b97e-4856-ac3f-96d38201688b","Type":"ContainerDied","Data":"76d34bb084e05745666ad9119dc63f53102b471918115e4cb6a4ba8964546fd8"}
Sep 17 18:41:50.149398 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.149372    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:41:50.159133 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.159096    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2s2x7\" (UniqueName: \"kubernetes.io/projected/47140786-7812-4af5-85c6-5528ce2bdd3a-kube-api-access-2s2x7\") pod \"47140786-7812-4af5-85c6-5528ce2bdd3a\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") "
Sep 17 18:41:50.159133 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.159137    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47140786-7812-4af5-85c6-5528ce2bdd3a-istio-data\") pod \"47140786-7812-4af5-85c6-5528ce2bdd3a\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") "
Sep 17 18:41:50.159347 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.159173    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47140786-7812-4af5-85c6-5528ce2bdd3a-istio-token\") pod \"47140786-7812-4af5-85c6-5528ce2bdd3a\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") "
Sep 17 18:41:50.159347 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.159290    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47140786-7812-4af5-85c6-5528ce2bdd3a-workload-certs\") pod \"47140786-7812-4af5-85c6-5528ce2bdd3a\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") "
Sep 17 18:41:50.159459 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.159391    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47140786-7812-4af5-85c6-5528ce2bdd3a-istio-envoy\") pod \"47140786-7812-4af5-85c6-5528ce2bdd3a\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") "
Sep 17 18:41:50.159459 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.159420    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47140786-7812-4af5-85c6-5528ce2bdd3a-credential-socket\") pod \"47140786-7812-4af5-85c6-5528ce2bdd3a\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") "
Sep 17 18:41:50.159579 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.159449    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47140786-7812-4af5-85c6-5528ce2bdd3a-istio-data" (OuterVolumeSpecName: "istio-data") pod "47140786-7812-4af5-85c6-5528ce2bdd3a" (UID: "47140786-7812-4af5-85c6-5528ce2bdd3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:41:50.159579 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.159478    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47140786-7812-4af5-85c6-5528ce2bdd3a-istiod-ca-cert\") pod \"47140786-7812-4af5-85c6-5528ce2bdd3a\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") "
Sep 17 18:41:50.159579 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.159481    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47140786-7812-4af5-85c6-5528ce2bdd3a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "47140786-7812-4af5-85c6-5528ce2bdd3a" (UID: "47140786-7812-4af5-85c6-5528ce2bdd3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:41:50.159579 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.159509    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47140786-7812-4af5-85c6-5528ce2bdd3a-istio-podinfo\") pod \"47140786-7812-4af5-85c6-5528ce2bdd3a\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") "
Sep 17 18:41:50.159579 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.159543    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47140786-7812-4af5-85c6-5528ce2bdd3a-workload-socket\") pod \"47140786-7812-4af5-85c6-5528ce2bdd3a\" (UID: \"47140786-7812-4af5-85c6-5528ce2bdd3a\") "
Sep 17 18:41:50.159865 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.159840    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47140786-7812-4af5-85c6-5528ce2bdd3a-istio-data\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:41:50.159865 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.159862    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47140786-7812-4af5-85c6-5528ce2bdd3a-workload-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:41:50.160003 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.159943    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47140786-7812-4af5-85c6-5528ce2bdd3a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "47140786-7812-4af5-85c6-5528ce2bdd3a" (UID: "47140786-7812-4af5-85c6-5528ce2bdd3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:41:50.160125 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.160100    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47140786-7812-4af5-85c6-5528ce2bdd3a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "47140786-7812-4af5-85c6-5528ce2bdd3a" (UID: "47140786-7812-4af5-85c6-5528ce2bdd3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:41:50.160269 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.160178    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47140786-7812-4af5-85c6-5528ce2bdd3a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "47140786-7812-4af5-85c6-5528ce2bdd3a" (UID: "47140786-7812-4af5-85c6-5528ce2bdd3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:41:50.161624 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.161597    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47140786-7812-4af5-85c6-5528ce2bdd3a-kube-api-access-2s2x7" (OuterVolumeSpecName: "kube-api-access-2s2x7") pod "47140786-7812-4af5-85c6-5528ce2bdd3a" (UID: "47140786-7812-4af5-85c6-5528ce2bdd3a"). InnerVolumeSpecName "kube-api-access-2s2x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:41:50.162072 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.162040    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47140786-7812-4af5-85c6-5528ce2bdd3a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "47140786-7812-4af5-85c6-5528ce2bdd3a" (UID: "47140786-7812-4af5-85c6-5528ce2bdd3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:41:50.162141 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.162105    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47140786-7812-4af5-85c6-5528ce2bdd3a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "47140786-7812-4af5-85c6-5528ce2bdd3a" (UID: "47140786-7812-4af5-85c6-5528ce2bdd3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 18:41:50.162263 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.162244    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47140786-7812-4af5-85c6-5528ce2bdd3a-istio-token" (OuterVolumeSpecName: "istio-token") pod "47140786-7812-4af5-85c6-5528ce2bdd3a" (UID: "47140786-7812-4af5-85c6-5528ce2bdd3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:41:50.260703 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.260654    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47140786-7812-4af5-85c6-5528ce2bdd3a-istio-envoy\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:41:50.260703 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.260694    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47140786-7812-4af5-85c6-5528ce2bdd3a-credential-socket\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:41:50.260703 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.260707    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47140786-7812-4af5-85c6-5528ce2bdd3a-istiod-ca-cert\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:41:50.261002 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.260720    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47140786-7812-4af5-85c6-5528ce2bdd3a-istio-podinfo\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:41:50.261002 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.260731    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47140786-7812-4af5-85c6-5528ce2bdd3a-workload-socket\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:41:50.261002 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.260742    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s2x7\" (UniqueName: \"kubernetes.io/projected/47140786-7812-4af5-85c6-5528ce2bdd3a-kube-api-access-2s2x7\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:41:50.261002 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.260755    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47140786-7812-4af5-85c6-5528ce2bdd3a-istio-token\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:41:50.337180 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.337146    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj" event={"ID":"b632c1dd-b97e-4856-ac3f-96d38201688b","Type":"ContainerStarted","Data":"03a62a3bef1e68c41e916860211f6ca0f4cef0de0c06603a83d940248839b3be"}
Sep 17 18:41:50.338786 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.338756    2581 generic.go:356] "Generic (PLEG): container finished" podID="47140786-7812-4af5-85c6-5528ce2bdd3a" containerID="767ae479458db4ede472c30d9657fa12063c94492326511272846a607806b0a2" exitCode=0
Sep 17 18:41:50.338944 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.338796    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"
Sep 17 18:41:50.338944 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.338808    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx" event={"ID":"47140786-7812-4af5-85c6-5528ce2bdd3a","Type":"ContainerDied","Data":"767ae479458db4ede472c30d9657fa12063c94492326511272846a607806b0a2"}
Sep 17 18:41:50.338944 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.338851    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx" event={"ID":"47140786-7812-4af5-85c6-5528ce2bdd3a","Type":"ContainerDied","Data":"5cc3c5c27c6022eb017cdeff7e6aafbda77f2661fa3e1ed1c9b700b1b39e40b2"}
Sep 17 18:41:50.338944 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.338875    2581 scope.go:117] "RemoveContainer" containerID="767ae479458db4ede472c30d9657fa12063c94492326511272846a607806b0a2"
Sep 17 18:41:50.349536 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.349512    2581 scope.go:117] "RemoveContainer" containerID="6d6782d230ec4bbfea00e673ee075729176d147fc746217d52abb8c615fa817e"
Sep 17 18:41:50.358664 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.358616    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj" podStartSLOduration=3.089792681 podStartE2EDuration="5.35859778s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.268805099s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:41:50.356592297 +0000 UTC m=+2698.447213952" watchObservedRunningTime="2026-09-17 18:41:50.35859778 +0000 UTC m=+2698.449219430"
Sep 17 18:41:50.360003 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.359583    2581 scope.go:117] "RemoveContainer" containerID="767ae479458db4ede472c30d9657fa12063c94492326511272846a607806b0a2"
Sep 17 18:41:50.363283 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:41:50.363249    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"767ae479458db4ede472c30d9657fa12063c94492326511272846a607806b0a2\": container with ID starting with 767ae479458db4ede472c30d9657fa12063c94492326511272846a607806b0a2 not found: ID does not exist" containerID="767ae479458db4ede472c30d9657fa12063c94492326511272846a607806b0a2"
Sep 17 18:41:50.363379 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.363295    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"767ae479458db4ede472c30d9657fa12063c94492326511272846a607806b0a2"} err="failed to get container status \"767ae479458db4ede472c30d9657fa12063c94492326511272846a607806b0a2\": rpc error: code = NotFound desc = could not find container \"767ae479458db4ede472c30d9657fa12063c94492326511272846a607806b0a2\": container with ID starting with 767ae479458db4ede472c30d9657fa12063c94492326511272846a607806b0a2 not found: ID does not exist"
Sep 17 18:41:50.363379 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.363319    2581 scope.go:117] "RemoveContainer" containerID="6d6782d230ec4bbfea00e673ee075729176d147fc746217d52abb8c615fa817e"
Sep 17 18:41:50.363961 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:41:50.363920    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d6782d230ec4bbfea00e673ee075729176d147fc746217d52abb8c615fa817e\": container with ID starting with 6d6782d230ec4bbfea00e673ee075729176d147fc746217d52abb8c615fa817e not found: ID does not exist" containerID="6d6782d230ec4bbfea00e673ee075729176d147fc746217d52abb8c615fa817e"
Sep 17 18:41:50.364115 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.364091    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d6782d230ec4bbfea00e673ee075729176d147fc746217d52abb8c615fa817e"} err="failed to get container status \"6d6782d230ec4bbfea00e673ee075729176d147fc746217d52abb8c615fa817e\": rpc error: code = NotFound desc = could not find container \"6d6782d230ec4bbfea00e673ee075729176d147fc746217d52abb8c615fa817e\": container with ID starting with 6d6782d230ec4bbfea00e673ee075729176d147fc746217d52abb8c615fa817e not found: ID does not exist"
Sep 17 18:41:50.374172 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.374144    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"]
Sep 17 18:41:50.379410 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.379352    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-cc7cb45dd-mcftx"]
Sep 17 18:41:50.679790 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:50.679700    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"
Sep 17 18:41:52.358049 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:52.358018    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47140786-7812-4af5-85c6-5528ce2bdd3a" path="/var/lib/kubelet/pods/47140786-7812-4af5-85c6-5528ce2bdd3a/volumes"
Sep 17 18:41:52.567298 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:52.567264    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/2.log"
Sep 17 18:41:52.578638 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:52.578613    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/2.log"
Sep 17 18:41:55.679859 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:55.679813    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"
Sep 17 18:41:55.681340 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:41:55.681310    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj" podUID="b632c1dd-b97e-4856-ac3f-96d38201688b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 17 18:42:05.680174 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:05.680125    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj" podUID="b632c1dd-b97e-4856-ac3f-96d38201688b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 17 18:42:12.153183 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:12.153147    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm"]
Sep 17 18:42:12.153661 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:12.153537    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47140786-7812-4af5-85c6-5528ce2bdd3a" containerName="istio-proxy"
Sep 17 18:42:12.153661 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:12.153550    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="47140786-7812-4af5-85c6-5528ce2bdd3a" containerName="istio-proxy"
Sep 17 18:42:12.153661 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:12.153565    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47140786-7812-4af5-85c6-5528ce2bdd3a" containerName="istio-proxy"
Sep 17 18:42:12.153661 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:12.153571    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="47140786-7812-4af5-85c6-5528ce2bdd3a" containerName="istio-proxy"
Sep 17 18:42:12.153661 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:12.153652    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="47140786-7812-4af5-85c6-5528ce2bdd3a" containerName="istio-proxy"
Sep 17 18:42:12.153661 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:12.153662    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="47140786-7812-4af5-85c6-5528ce2bdd3a" containerName="istio-proxy"
Sep 17 18:42:12.156634 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:12.156615    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm"
Sep 17 18:42:12.160504 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:12.160477    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-xsdbq\""
Sep 17 18:42:12.160633 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:12.160489    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-zhdlm\""
Sep 17 18:42:12.160699 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:12.160631    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 17 18:42:12.160699 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:12.160654    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 17 18:42:12.160881 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:12.160857    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 17 18:42:12.164572 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:12.164545    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm"]
Sep 17 18:42:12.270435 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:12.270372    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7g84\" (UniqueName: \"kubernetes.io/projected/b424313e-9217-424d-a9e4-c8b3f51d28d9-kube-api-access-m7g84\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm\" (UID: \"b424313e-9217-424d-a9e4-c8b3f51d28d9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm"
Sep 17 18:42:12.270435 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:12.270431    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b424313e-9217-424d-a9e4-c8b3f51d28d9-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm\" (UID: \"b424313e-9217-424d-a9e4-c8b3f51d28d9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm"
Sep 17 18:42:12.371140 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:12.371090    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b424313e-9217-424d-a9e4-c8b3f51d28d9-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm\" (UID: \"b424313e-9217-424d-a9e4-c8b3f51d28d9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm"
Sep 17 18:42:12.371360 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:12.371201    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7g84\" (UniqueName: \"kubernetes.io/projected/b424313e-9217-424d-a9e4-c8b3f51d28d9-kube-api-access-m7g84\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm\" (UID: \"b424313e-9217-424d-a9e4-c8b3f51d28d9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm"
Sep 17 18:42:12.373770 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:12.373736    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b424313e-9217-424d-a9e4-c8b3f51d28d9-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm\" (UID: \"b424313e-9217-424d-a9e4-c8b3f51d28d9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm"
Sep 17 18:42:12.379618 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:12.379588    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7g84\" (UniqueName: \"kubernetes.io/projected/b424313e-9217-424d-a9e4-c8b3f51d28d9-kube-api-access-m7g84\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm\" (UID: \"b424313e-9217-424d-a9e4-c8b3f51d28d9\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm"
Sep 17 18:42:12.468322 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:12.468280    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm"
Sep 17 18:42:12.744781 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:12.744751    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm"]
Sep 17 18:42:12.747615 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:42:12.747584    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb424313e_9217_424d_a9e4_c8b3f51d28d9.slice/crio-5fd698195fe3f0bcc4b578c9fc0ac3419025ed32781995bac8e917f8fc256343 WatchSource:0}: Error finding container 5fd698195fe3f0bcc4b578c9fc0ac3419025ed32781995bac8e917f8fc256343: Status 404 returned error can't find the container with id 5fd698195fe3f0bcc4b578c9fc0ac3419025ed32781995bac8e917f8fc256343
Sep 17 18:42:13.454235 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:13.454187    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm" event={"ID":"b424313e-9217-424d-a9e4-c8b3f51d28d9","Type":"ContainerStarted","Data":"200ccdd45a2c1619d00b95834101279849889ac59bb95ae6e704d4cff8958eab"}
Sep 17 18:42:13.454656 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:13.454254    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm"
Sep 17 18:42:13.454656 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:13.454278    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm" event={"ID":"b424313e-9217-424d-a9e4-c8b3f51d28d9","Type":"ContainerStarted","Data":"5fd698195fe3f0bcc4b578c9fc0ac3419025ed32781995bac8e917f8fc256343"}
Sep 17 18:42:13.470657 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:13.470588    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm" podStartSLOduration=1.470568437 podStartE2EDuration="1.470568437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:42:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:42:13.470522908 +0000 UTC m=+2721.561144554" watchObservedRunningTime="2026-09-17 18:42:13.470568437 +0000 UTC m=+2721.561190089"
Sep 17 18:42:15.680177 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:15.680132    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj" podUID="b632c1dd-b97e-4856-ac3f-96d38201688b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 17 18:42:17.320786 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:17.320752    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l"
Sep 17 18:42:25.680346 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:25.680288    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj" podUID="b632c1dd-b97e-4856-ac3f-96d38201688b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 17 18:42:35.679637 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:35.679593    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj" podUID="b632c1dd-b97e-4856-ac3f-96d38201688b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 17 18:42:44.464129 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:44.464097    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm"
Sep 17 18:42:45.679548 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:45.679505    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj" podUID="b632c1dd-b97e-4856-ac3f-96d38201688b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 17 18:42:55.679661 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:42:55.679617    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj" podUID="b632c1dd-b97e-4856-ac3f-96d38201688b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 17 18:43:05.679677 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:43:05.679626    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj" podUID="b632c1dd-b97e-4856-ac3f-96d38201688b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 17 18:43:15.680012 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:43:15.679918    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj" podUID="b632c1dd-b97e-4856-ac3f-96d38201688b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 17 18:43:25.679743 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:43:25.679694    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj" podUID="b632c1dd-b97e-4856-ac3f-96d38201688b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 17 18:43:35.679970 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:43:35.679891    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj" podUID="b632c1dd-b97e-4856-ac3f-96d38201688b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 17 18:43:45.689537 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:43:45.689504    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"
Sep 17 18:43:45.697187 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:43:45.697157    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"
Sep 17 18:43:47.381992 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:43:47.381954    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"]
Sep 17 18:43:47.384628 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:43:47.384598    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l"]
Sep 17 18:43:57.321343 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:43:57.321300    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l" podUID="612a768e-20be-4759-a1b9-15611fd9d28e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:44:02.406983 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:02.406935    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj" podUID="b632c1dd-b97e-4856-ac3f-96d38201688b" containerName="main" containerID="cri-o://03a62a3bef1e68c41e916860211f6ca0f4cef0de0c06603a83d940248839b3be" gracePeriod=45
Sep 17 18:44:02.413258 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:02.413211    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l" podUID="612a768e-20be-4759-a1b9-15611fd9d28e" containerName="main" containerID="cri-o://05c2d8de8182c8b07df5317d933fe5fdbdcecf9a73d0bcc5facd7aa1a295cf76" gracePeriod=45
Sep 17 18:44:02.690918 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:02.690801    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj" podUID="b632c1dd-b97e-4856-ac3f-96d38201688b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Sep 17 18:44:03.165210 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:03.165182    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"
Sep 17 18:44:03.203954 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:03.203919    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b632c1dd-b97e-4856-ac3f-96d38201688b-home\") pod \"b632c1dd-b97e-4856-ac3f-96d38201688b\" (UID: \"b632c1dd-b97e-4856-ac3f-96d38201688b\") "
Sep 17 18:44:03.204141 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:03.203974    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98h7q\" (UniqueName: \"kubernetes.io/projected/b632c1dd-b97e-4856-ac3f-96d38201688b-kube-api-access-98h7q\") pod \"b632c1dd-b97e-4856-ac3f-96d38201688b\" (UID: \"b632c1dd-b97e-4856-ac3f-96d38201688b\") "
Sep 17 18:44:03.204141 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:03.204012    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b632c1dd-b97e-4856-ac3f-96d38201688b-model-cache\") pod \"b632c1dd-b97e-4856-ac3f-96d38201688b\" (UID: \"b632c1dd-b97e-4856-ac3f-96d38201688b\") "
Sep 17 18:44:03.204141 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:03.204048    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b632c1dd-b97e-4856-ac3f-96d38201688b-tmp-dir\") pod \"b632c1dd-b97e-4856-ac3f-96d38201688b\" (UID: \"b632c1dd-b97e-4856-ac3f-96d38201688b\") "
Sep 17 18:44:03.204141 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:03.204086    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b632c1dd-b97e-4856-ac3f-96d38201688b-kserve-provision-location\") pod \"b632c1dd-b97e-4856-ac3f-96d38201688b\" (UID: \"b632c1dd-b97e-4856-ac3f-96d38201688b\") "
Sep 17 18:44:03.204389 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:03.204151    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b632c1dd-b97e-4856-ac3f-96d38201688b-dshm\") pod \"b632c1dd-b97e-4856-ac3f-96d38201688b\" (UID: \"b632c1dd-b97e-4856-ac3f-96d38201688b\") "
Sep 17 18:44:03.204389 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:03.204196    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b632c1dd-b97e-4856-ac3f-96d38201688b-cabundle-cert\") pod \"b632c1dd-b97e-4856-ac3f-96d38201688b\" (UID: \"b632c1dd-b97e-4856-ac3f-96d38201688b\") "
Sep 17 18:44:03.204389 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:03.204237    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b632c1dd-b97e-4856-ac3f-96d38201688b-tls-certs\") pod \"b632c1dd-b97e-4856-ac3f-96d38201688b\" (UID: \"b632c1dd-b97e-4856-ac3f-96d38201688b\") "
Sep 17 18:44:03.204557 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:03.204443    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b632c1dd-b97e-4856-ac3f-96d38201688b-model-cache" (OuterVolumeSpecName: "model-cache") pod "b632c1dd-b97e-4856-ac3f-96d38201688b" (UID: "b632c1dd-b97e-4856-ac3f-96d38201688b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:44:03.204610 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:03.204596    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b632c1dd-b97e-4856-ac3f-96d38201688b-model-cache\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:44:03.204780 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:03.204732    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b632c1dd-b97e-4856-ac3f-96d38201688b-home" (OuterVolumeSpecName: "home") pod "b632c1dd-b97e-4856-ac3f-96d38201688b" (UID: "b632c1dd-b97e-4856-ac3f-96d38201688b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:44:03.204961 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:03.204939    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b632c1dd-b97e-4856-ac3f-96d38201688b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b632c1dd-b97e-4856-ac3f-96d38201688b" (UID: "b632c1dd-b97e-4856-ac3f-96d38201688b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:44:03.207082 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:03.207054    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b632c1dd-b97e-4856-ac3f-96d38201688b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b632c1dd-b97e-4856-ac3f-96d38201688b" (UID: "b632c1dd-b97e-4856-ac3f-96d38201688b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:44:03.207251 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:03.207225    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b632c1dd-b97e-4856-ac3f-96d38201688b-kube-api-access-98h7q" (OuterVolumeSpecName: "kube-api-access-98h7q") pod "b632c1dd-b97e-4856-ac3f-96d38201688b" (UID: "b632c1dd-b97e-4856-ac3f-96d38201688b"). InnerVolumeSpecName "kube-api-access-98h7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:44:03.207342 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:03.207293    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b632c1dd-b97e-4856-ac3f-96d38201688b-dshm" (OuterVolumeSpecName: "dshm") pod "b632c1dd-b97e-4856-ac3f-96d38201688b" (UID: "b632c1dd-b97e-4856-ac3f-96d38201688b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:44:03.224291 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:03.224250    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b632c1dd-b97e-4856-ac3f-96d38201688b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b632c1dd-b97e-4856-ac3f-96d38201688b" (UID: "b632c1dd-b97e-4856-ac3f-96d38201688b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:44:03.232216 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:03.232178    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b632c1dd-b97e-4856-ac3f-96d38201688b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b632c1dd-b97e-4856-ac3f-96d38201688b" (UID: "b632c1dd-b97e-4856-ac3f-96d38201688b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:44:03.305807 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:03.305771    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b632c1dd-b97e-4856-ac3f-96d38201688b-tmp-dir\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:44:03.305807 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:03.305803    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b632c1dd-b97e-4856-ac3f-96d38201688b-kserve-provision-location\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:44:03.305807 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:03.305814    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b632c1dd-b97e-4856-ac3f-96d38201688b-dshm\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:44:03.306087 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:03.305823    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b632c1dd-b97e-4856-ac3f-96d38201688b-cabundle-cert\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:44:03.306087 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:03.305831    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b632c1dd-b97e-4856-ac3f-96d38201688b-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:44:03.306087 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:03.305839    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b632c1dd-b97e-4856-ac3f-96d38201688b-home\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:44:03.306087 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:03.305847    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98h7q\" (UniqueName: \"kubernetes.io/projected/b632c1dd-b97e-4856-ac3f-96d38201688b-kube-api-access-98h7q\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:44:04.008108 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:04.008064    2581 generic.go:356] "Generic (PLEG): container finished" podID="b632c1dd-b97e-4856-ac3f-96d38201688b" containerID="03a62a3bef1e68c41e916860211f6ca0f4cef0de0c06603a83d940248839b3be" exitCode=0
Sep 17 18:44:04.008535 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:04.008142    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"
Sep 17 18:44:04.008535 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:04.008135    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj" event={"ID":"b632c1dd-b97e-4856-ac3f-96d38201688b","Type":"ContainerDied","Data":"03a62a3bef1e68c41e916860211f6ca0f4cef0de0c06603a83d940248839b3be"}
Sep 17 18:44:04.008535 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:04.008269    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj" event={"ID":"b632c1dd-b97e-4856-ac3f-96d38201688b","Type":"ContainerDied","Data":"38e53794f8c7ee5920df16c4188f07d2652b9e8d241c49172b296e36592ce909"}
Sep 17 18:44:04.008535 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:04.008292    2581 scope.go:117] "RemoveContainer" containerID="03a62a3bef1e68c41e916860211f6ca0f4cef0de0c06603a83d940248839b3be"
Sep 17 18:44:04.017607 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:04.017584    2581 scope.go:117] "RemoveContainer" containerID="76d34bb084e05745666ad9119dc63f53102b471918115e4cb6a4ba8964546fd8"
Sep 17 18:44:04.026311 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:04.026289    2581 scope.go:117] "RemoveContainer" containerID="03a62a3bef1e68c41e916860211f6ca0f4cef0de0c06603a83d940248839b3be"
Sep 17 18:44:04.026622 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:44:04.026599    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03a62a3bef1e68c41e916860211f6ca0f4cef0de0c06603a83d940248839b3be\": container with ID starting with 03a62a3bef1e68c41e916860211f6ca0f4cef0de0c06603a83d940248839b3be not found: ID does not exist" containerID="03a62a3bef1e68c41e916860211f6ca0f4cef0de0c06603a83d940248839b3be"
Sep 17 18:44:04.026700 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:04.026632    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03a62a3bef1e68c41e916860211f6ca0f4cef0de0c06603a83d940248839b3be"} err="failed to get container status \"03a62a3bef1e68c41e916860211f6ca0f4cef0de0c06603a83d940248839b3be\": rpc error: code = NotFound desc = could not find container \"03a62a3bef1e68c41e916860211f6ca0f4cef0de0c06603a83d940248839b3be\": container with ID starting with 03a62a3bef1e68c41e916860211f6ca0f4cef0de0c06603a83d940248839b3be not found: ID does not exist"
Sep 17 18:44:04.026700 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:04.026652    2581 scope.go:117] "RemoveContainer" containerID="76d34bb084e05745666ad9119dc63f53102b471918115e4cb6a4ba8964546fd8"
Sep 17 18:44:04.026942 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:44:04.026892    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76d34bb084e05745666ad9119dc63f53102b471918115e4cb6a4ba8964546fd8\": container with ID starting with 76d34bb084e05745666ad9119dc63f53102b471918115e4cb6a4ba8964546fd8 not found: ID does not exist" containerID="76d34bb084e05745666ad9119dc63f53102b471918115e4cb6a4ba8964546fd8"
Sep 17 18:44:04.026989 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:04.026950    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76d34bb084e05745666ad9119dc63f53102b471918115e4cb6a4ba8964546fd8"} err="failed to get container status \"76d34bb084e05745666ad9119dc63f53102b471918115e4cb6a4ba8964546fd8\": rpc error: code = NotFound desc = could not find container \"76d34bb084e05745666ad9119dc63f53102b471918115e4cb6a4ba8964546fd8\": container with ID starting with 76d34bb084e05745666ad9119dc63f53102b471918115e4cb6a4ba8964546fd8 not found: ID does not exist"
Sep 17 18:44:04.030369 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:04.030343    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"]
Sep 17 18:44:04.035190 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:04.035164    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-66fd6b68fd-h5dlj"]
Sep 17 18:44:04.357761 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:04.357678    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b632c1dd-b97e-4856-ac3f-96d38201688b" path="/var/lib/kubelet/pods/b632c1dd-b97e-4856-ac3f-96d38201688b/volumes"
Sep 17 18:44:07.320601 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:07.320554    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l" podUID="612a768e-20be-4759-a1b9-15611fd9d28e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:44:17.321304 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:17.321261    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l" podUID="612a768e-20be-4759-a1b9-15611fd9d28e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:44:17.321715 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:17.321377    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l"
Sep 17 18:44:27.320604 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:27.320565    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l" podUID="612a768e-20be-4759-a1b9-15611fd9d28e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:44:32.649909 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:32.649877    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l"
Sep 17 18:44:32.780675 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:32.780620    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/612a768e-20be-4759-a1b9-15611fd9d28e-tls-certs\") pod \"612a768e-20be-4759-a1b9-15611fd9d28e\" (UID: \"612a768e-20be-4759-a1b9-15611fd9d28e\") "
Sep 17 18:44:32.780876 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:32.780731    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccxsx\" (UniqueName: \"kubernetes.io/projected/612a768e-20be-4759-a1b9-15611fd9d28e-kube-api-access-ccxsx\") pod \"612a768e-20be-4759-a1b9-15611fd9d28e\" (UID: \"612a768e-20be-4759-a1b9-15611fd9d28e\") "
Sep 17 18:44:32.782766 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:32.782732    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/612a768e-20be-4759-a1b9-15611fd9d28e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "612a768e-20be-4759-a1b9-15611fd9d28e" (UID: "612a768e-20be-4759-a1b9-15611fd9d28e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:44:32.782766 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:32.782754    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/612a768e-20be-4759-a1b9-15611fd9d28e-kube-api-access-ccxsx" (OuterVolumeSpecName: "kube-api-access-ccxsx") pod "612a768e-20be-4759-a1b9-15611fd9d28e" (UID: "612a768e-20be-4759-a1b9-15611fd9d28e"). InnerVolumeSpecName "kube-api-access-ccxsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:44:32.882050 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:32.882011    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccxsx\" (UniqueName: \"kubernetes.io/projected/612a768e-20be-4759-a1b9-15611fd9d28e-kube-api-access-ccxsx\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:44:32.882050 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:32.882041    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/612a768e-20be-4759-a1b9-15611fd9d28e-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:44:33.146273 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:33.146181    2581 generic.go:356] "Generic (PLEG): container finished" podID="612a768e-20be-4759-a1b9-15611fd9d28e" containerID="05c2d8de8182c8b07df5317d933fe5fdbdcecf9a73d0bcc5facd7aa1a295cf76" exitCode=0
Sep 17 18:44:33.146273 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:33.146241    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l"
Sep 17 18:44:33.146477 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:33.146273    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l" event={"ID":"612a768e-20be-4759-a1b9-15611fd9d28e","Type":"ContainerDied","Data":"05c2d8de8182c8b07df5317d933fe5fdbdcecf9a73d0bcc5facd7aa1a295cf76"}
Sep 17 18:44:33.146477 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:33.146324    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l" event={"ID":"612a768e-20be-4759-a1b9-15611fd9d28e","Type":"ContainerDied","Data":"b33789d9c1f9e5fcac62122090a15ff549630782b74fab99021d06f706c176fd"}
Sep 17 18:44:33.146477 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:33.146349    2581 scope.go:117] "RemoveContainer" containerID="05c2d8de8182c8b07df5317d933fe5fdbdcecf9a73d0bcc5facd7aa1a295cf76"
Sep 17 18:44:33.156672 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:33.156642    2581 scope.go:117] "RemoveContainer" containerID="05c2d8de8182c8b07df5317d933fe5fdbdcecf9a73d0bcc5facd7aa1a295cf76"
Sep 17 18:44:33.156922 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:44:33.156880    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05c2d8de8182c8b07df5317d933fe5fdbdcecf9a73d0bcc5facd7aa1a295cf76\": container with ID starting with 05c2d8de8182c8b07df5317d933fe5fdbdcecf9a73d0bcc5facd7aa1a295cf76 not found: ID does not exist" containerID="05c2d8de8182c8b07df5317d933fe5fdbdcecf9a73d0bcc5facd7aa1a295cf76"
Sep 17 18:44:33.157011 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:33.156925    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05c2d8de8182c8b07df5317d933fe5fdbdcecf9a73d0bcc5facd7aa1a295cf76"} err="failed to get container status \"05c2d8de8182c8b07df5317d933fe5fdbdcecf9a73d0bcc5facd7aa1a295cf76\": rpc error: code = NotFound desc = could not find container \"05c2d8de8182c8b07df5317d933fe5fdbdcecf9a73d0bcc5facd7aa1a295cf76\": container with ID starting with 05c2d8de8182c8b07df5317d933fe5fdbdcecf9a73d0bcc5facd7aa1a295cf76 not found: ID does not exist"
Sep 17 18:44:33.168540 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:33.168514    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l"]
Sep 17 18:44:33.172210 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:33.172185    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-w4h9l"]
Sep 17 18:44:34.358875 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:34.358842    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="612a768e-20be-4759-a1b9-15611fd9d28e" path="/var/lib/kubelet/pods/612a768e-20be-4759-a1b9-15611fd9d28e/volumes"
Sep 17 18:44:47.584423 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.584208    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"]
Sep 17 18:44:47.585118 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.585095    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b632c1dd-b97e-4856-ac3f-96d38201688b" containerName="main"
Sep 17 18:44:47.585213 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.585122    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b632c1dd-b97e-4856-ac3f-96d38201688b" containerName="main"
Sep 17 18:44:47.585213 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.585145    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="612a768e-20be-4759-a1b9-15611fd9d28e" containerName="main"
Sep 17 18:44:47.585213 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.585153    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="612a768e-20be-4759-a1b9-15611fd9d28e" containerName="main"
Sep 17 18:44:47.585213 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.585175    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b632c1dd-b97e-4856-ac3f-96d38201688b" containerName="storage-initializer"
Sep 17 18:44:47.585213 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.585185    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b632c1dd-b97e-4856-ac3f-96d38201688b" containerName="storage-initializer"
Sep 17 18:44:47.585467 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.585332    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="612a768e-20be-4759-a1b9-15611fd9d28e" containerName="main"
Sep 17 18:44:47.585467 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.585346    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b632c1dd-b97e-4856-ac3f-96d38201688b" containerName="main"
Sep 17 18:44:47.589817 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.589796    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"
Sep 17 18:44:47.592269 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.592243    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Sep 17 18:44:47.592269 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.592259    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Sep 17 18:44:47.592460 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.592374    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Sep 17 18:44:47.592685 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.592663    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-dd65c\""
Sep 17 18:44:47.592779 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.592685    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Sep 17 18:44:47.593376 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.593359    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:44:47.601609 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.601580    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"]
Sep 17 18:44:47.615617 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.615587    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32c04d1d-a975-48b5-9161-25dc30bc5702-cabundle-cert\") pod \"lora-single-adapter-test-kserve-9b6dc798d-57sjm\" (UID: \"32c04d1d-a975-48b5-9161-25dc30bc5702\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"
Sep 17 18:44:47.615766 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.615638    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32c04d1d-a975-48b5-9161-25dc30bc5702-model-cache\") pod \"lora-single-adapter-test-kserve-9b6dc798d-57sjm\" (UID: \"32c04d1d-a975-48b5-9161-25dc30bc5702\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"
Sep 17 18:44:47.615766 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.615658    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32c04d1d-a975-48b5-9161-25dc30bc5702-tmp-dir\") pod \"lora-single-adapter-test-kserve-9b6dc798d-57sjm\" (UID: \"32c04d1d-a975-48b5-9161-25dc30bc5702\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"
Sep 17 18:44:47.615766 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.615733    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32c04d1d-a975-48b5-9161-25dc30bc5702-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-9b6dc798d-57sjm\" (UID: \"32c04d1d-a975-48b5-9161-25dc30bc5702\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"
Sep 17 18:44:47.615949 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.615773    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqb9n\" (UniqueName: \"kubernetes.io/projected/32c04d1d-a975-48b5-9161-25dc30bc5702-kube-api-access-vqb9n\") pod \"lora-single-adapter-test-kserve-9b6dc798d-57sjm\" (UID: \"32c04d1d-a975-48b5-9161-25dc30bc5702\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"
Sep 17 18:44:47.615949 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.615809    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32c04d1d-a975-48b5-9161-25dc30bc5702-dshm\") pod \"lora-single-adapter-test-kserve-9b6dc798d-57sjm\" (UID: \"32c04d1d-a975-48b5-9161-25dc30bc5702\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"
Sep 17 18:44:47.615949 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.615837    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32c04d1d-a975-48b5-9161-25dc30bc5702-tls-certs\") pod \"lora-single-adapter-test-kserve-9b6dc798d-57sjm\" (UID: \"32c04d1d-a975-48b5-9161-25dc30bc5702\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"
Sep 17 18:44:47.615949 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.615937    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32c04d1d-a975-48b5-9161-25dc30bc5702-home\") pod \"lora-single-adapter-test-kserve-9b6dc798d-57sjm\" (UID: \"32c04d1d-a975-48b5-9161-25dc30bc5702\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"
Sep 17 18:44:47.716825 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.716780    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32c04d1d-a975-48b5-9161-25dc30bc5702-dshm\") pod \"lora-single-adapter-test-kserve-9b6dc798d-57sjm\" (UID: \"32c04d1d-a975-48b5-9161-25dc30bc5702\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"
Sep 17 18:44:47.716825 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.716826    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32c04d1d-a975-48b5-9161-25dc30bc5702-tls-certs\") pod \"lora-single-adapter-test-kserve-9b6dc798d-57sjm\" (UID: \"32c04d1d-a975-48b5-9161-25dc30bc5702\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"
Sep 17 18:44:47.717128 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.716880    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32c04d1d-a975-48b5-9161-25dc30bc5702-home\") pod \"lora-single-adapter-test-kserve-9b6dc798d-57sjm\" (UID: \"32c04d1d-a975-48b5-9161-25dc30bc5702\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"
Sep 17 18:44:47.717128 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.717054    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32c04d1d-a975-48b5-9161-25dc30bc5702-cabundle-cert\") pod \"lora-single-adapter-test-kserve-9b6dc798d-57sjm\" (UID: \"32c04d1d-a975-48b5-9161-25dc30bc5702\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"
Sep 17 18:44:47.717242 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.717144    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32c04d1d-a975-48b5-9161-25dc30bc5702-model-cache\") pod \"lora-single-adapter-test-kserve-9b6dc798d-57sjm\" (UID: \"32c04d1d-a975-48b5-9161-25dc30bc5702\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"
Sep 17 18:44:47.717242 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.717181    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32c04d1d-a975-48b5-9161-25dc30bc5702-tmp-dir\") pod \"lora-single-adapter-test-kserve-9b6dc798d-57sjm\" (UID: \"32c04d1d-a975-48b5-9161-25dc30bc5702\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"
Sep 17 18:44:47.717385 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.717345    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32c04d1d-a975-48b5-9161-25dc30bc5702-home\") pod \"lora-single-adapter-test-kserve-9b6dc798d-57sjm\" (UID: \"32c04d1d-a975-48b5-9161-25dc30bc5702\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"
Sep 17 18:44:47.717501 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.717237    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32c04d1d-a975-48b5-9161-25dc30bc5702-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-9b6dc798d-57sjm\" (UID: \"32c04d1d-a975-48b5-9161-25dc30bc5702\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"
Sep 17 18:44:47.717637 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.717607    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqb9n\" (UniqueName: \"kubernetes.io/projected/32c04d1d-a975-48b5-9161-25dc30bc5702-kube-api-access-vqb9n\") pod \"lora-single-adapter-test-kserve-9b6dc798d-57sjm\" (UID: \"32c04d1d-a975-48b5-9161-25dc30bc5702\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"
Sep 17 18:44:47.717848 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.717810    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32c04d1d-a975-48b5-9161-25dc30bc5702-cabundle-cert\") pod \"lora-single-adapter-test-kserve-9b6dc798d-57sjm\" (UID: \"32c04d1d-a975-48b5-9161-25dc30bc5702\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"
Sep 17 18:44:47.718338 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.718208    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32c04d1d-a975-48b5-9161-25dc30bc5702-model-cache\") pod \"lora-single-adapter-test-kserve-9b6dc798d-57sjm\" (UID: \"32c04d1d-a975-48b5-9161-25dc30bc5702\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"
Sep 17 18:44:47.718452 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.718373    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32c04d1d-a975-48b5-9161-25dc30bc5702-tmp-dir\") pod \"lora-single-adapter-test-kserve-9b6dc798d-57sjm\" (UID: \"32c04d1d-a975-48b5-9161-25dc30bc5702\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"
Sep 17 18:44:47.718517 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.718468    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32c04d1d-a975-48b5-9161-25dc30bc5702-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-9b6dc798d-57sjm\" (UID: \"32c04d1d-a975-48b5-9161-25dc30bc5702\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"
Sep 17 18:44:47.719696 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.719669    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32c04d1d-a975-48b5-9161-25dc30bc5702-dshm\") pod \"lora-single-adapter-test-kserve-9b6dc798d-57sjm\" (UID: \"32c04d1d-a975-48b5-9161-25dc30bc5702\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"
Sep 17 18:44:47.720011 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.719994    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32c04d1d-a975-48b5-9161-25dc30bc5702-tls-certs\") pod \"lora-single-adapter-test-kserve-9b6dc798d-57sjm\" (UID: \"32c04d1d-a975-48b5-9161-25dc30bc5702\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"
Sep 17 18:44:47.725659 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.725637    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqb9n\" (UniqueName: \"kubernetes.io/projected/32c04d1d-a975-48b5-9161-25dc30bc5702-kube-api-access-vqb9n\") pod \"lora-single-adapter-test-kserve-9b6dc798d-57sjm\" (UID: \"32c04d1d-a975-48b5-9161-25dc30bc5702\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"
Sep 17 18:44:47.901194 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:47.901095    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"
Sep 17 18:44:48.180024 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:48.179987    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"]
Sep 17 18:44:48.181391 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:44:48.181365    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32c04d1d_a975_48b5_9161_25dc30bc5702.slice/crio-a7b2e95c933c07891be5c2ef2a5b7163b0f234f03c8e71c8441eb13ae9effaf9 WatchSource:0}: Error finding container a7b2e95c933c07891be5c2ef2a5b7163b0f234f03c8e71c8441eb13ae9effaf9: Status 404 returned error can't find the container with id a7b2e95c933c07891be5c2ef2a5b7163b0f234f03c8e71c8441eb13ae9effaf9
Sep 17 18:44:48.218715 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:48.218678    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm" event={"ID":"32c04d1d-a975-48b5-9161-25dc30bc5702","Type":"ContainerStarted","Data":"a7b2e95c933c07891be5c2ef2a5b7163b0f234f03c8e71c8441eb13ae9effaf9"}
Sep 17 18:44:49.228221 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:49.228171    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm" event={"ID":"32c04d1d-a975-48b5-9161-25dc30bc5702","Type":"ContainerStarted","Data":"67b0ec7b951fa67d18e7df728bbe6606de472ff8e880a902710e215913fdd2e3"}
Sep 17 18:44:51.240493 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:51.240454    2581 generic.go:356] "Generic (PLEG): container finished" podID="32c04d1d-a975-48b5-9161-25dc30bc5702" containerID="67b0ec7b951fa67d18e7df728bbe6606de472ff8e880a902710e215913fdd2e3" exitCode=0
Sep 17 18:44:51.240493 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:51.240485    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm" event={"ID":"32c04d1d-a975-48b5-9161-25dc30bc5702","Type":"ContainerDied","Data":"67b0ec7b951fa67d18e7df728bbe6606de472ff8e880a902710e215913fdd2e3"}
Sep 17 18:44:52.248625 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:52.248587    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm" event={"ID":"32c04d1d-a975-48b5-9161-25dc30bc5702","Type":"ContainerStarted","Data":"d8744100e43a250da2fc84a85adffb68e03585cd6604c084f7895e3943ad9c47"}
Sep 17 18:44:52.272240 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:52.272170    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm" podStartSLOduration=2.508161925 podStartE2EDuration="5.272147253s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.763985328s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:44:52.269411784 +0000 UTC m=+2880.360033447" watchObservedRunningTime="2026-09-17 18:44:52.272147253 +0000 UTC m=+2880.362768904"
Sep 17 18:44:52.901581 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:52.901539    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"
Sep 17 18:44:57.902209 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:57.902171    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"
Sep 17 18:44:57.903763 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:44:57.903731    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm" podUID="32c04d1d-a975-48b5-9161-25dc30bc5702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 17 18:45:03.937186 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:03.937145    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm"]
Sep 17 18:45:04.464207 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:04.464160    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm" podUID="b424313e-9217-424d-a9e4-c8b3f51d28d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:45:07.902310 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:07.902261    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm" podUID="32c04d1d-a975-48b5-9161-25dc30bc5702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 17 18:45:14.463887 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:14.463842    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm" podUID="b424313e-9217-424d-a9e4-c8b3f51d28d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:45:17.902241 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:17.902184    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm" podUID="32c04d1d-a975-48b5-9161-25dc30bc5702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 17 18:45:18.956684 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:18.956635    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm" podUID="b424313e-9217-424d-a9e4-c8b3f51d28d9" containerName="main" containerID="cri-o://200ccdd45a2c1619d00b95834101279849889ac59bb95ae6e704d4cff8958eab" gracePeriod=45
Sep 17 18:45:24.463369 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:24.463329    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm" podUID="b424313e-9217-424d-a9e4-c8b3f51d28d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:45:24.463816 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:24.463458    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm"
Sep 17 18:45:27.901616 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:27.901567    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm" podUID="32c04d1d-a975-48b5-9161-25dc30bc5702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 17 18:45:34.464002 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:34.463959    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm" podUID="b424313e-9217-424d-a9e4-c8b3f51d28d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:45:37.902128 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:37.902072    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm" podUID="32c04d1d-a975-48b5-9161-25dc30bc5702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 17 18:45:44.463672 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:44.463632    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm" podUID="b424313e-9217-424d-a9e4-c8b3f51d28d9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:45:47.902545 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:47.902490    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm" podUID="32c04d1d-a975-48b5-9161-25dc30bc5702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 17 18:45:49.209486 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:49.209455    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm"
Sep 17 18:45:49.292871 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:49.292771    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7g84\" (UniqueName: \"kubernetes.io/projected/b424313e-9217-424d-a9e4-c8b3f51d28d9-kube-api-access-m7g84\") pod \"b424313e-9217-424d-a9e4-c8b3f51d28d9\" (UID: \"b424313e-9217-424d-a9e4-c8b3f51d28d9\") "
Sep 17 18:45:49.293087 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:49.292890    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b424313e-9217-424d-a9e4-c8b3f51d28d9-tls-certs\") pod \"b424313e-9217-424d-a9e4-c8b3f51d28d9\" (UID: \"b424313e-9217-424d-a9e4-c8b3f51d28d9\") "
Sep 17 18:45:49.295142 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:49.295113    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b424313e-9217-424d-a9e4-c8b3f51d28d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b424313e-9217-424d-a9e4-c8b3f51d28d9" (UID: "b424313e-9217-424d-a9e4-c8b3f51d28d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:45:49.295142 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:49.295121    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b424313e-9217-424d-a9e4-c8b3f51d28d9-kube-api-access-m7g84" (OuterVolumeSpecName: "kube-api-access-m7g84") pod "b424313e-9217-424d-a9e4-c8b3f51d28d9" (UID: "b424313e-9217-424d-a9e4-c8b3f51d28d9"). InnerVolumeSpecName "kube-api-access-m7g84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:45:49.393786 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:49.393743    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7g84\" (UniqueName: \"kubernetes.io/projected/b424313e-9217-424d-a9e4-c8b3f51d28d9-kube-api-access-m7g84\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:45:49.393786 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:49.393781    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b424313e-9217-424d-a9e4-c8b3f51d28d9-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:45:49.543058 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:49.542967    2581 generic.go:356] "Generic (PLEG): container finished" podID="b424313e-9217-424d-a9e4-c8b3f51d28d9" containerID="200ccdd45a2c1619d00b95834101279849889ac59bb95ae6e704d4cff8958eab" exitCode=0
Sep 17 18:45:49.543058 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:49.543017    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm"
Sep 17 18:45:49.543240 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:49.543063    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm" event={"ID":"b424313e-9217-424d-a9e4-c8b3f51d28d9","Type":"ContainerDied","Data":"200ccdd45a2c1619d00b95834101279849889ac59bb95ae6e704d4cff8958eab"}
Sep 17 18:45:49.543240 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:49.543128    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm" event={"ID":"b424313e-9217-424d-a9e4-c8b3f51d28d9","Type":"ContainerDied","Data":"5fd698195fe3f0bcc4b578c9fc0ac3419025ed32781995bac8e917f8fc256343"}
Sep 17 18:45:49.543240 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:49.543161    2581 scope.go:117] "RemoveContainer" containerID="200ccdd45a2c1619d00b95834101279849889ac59bb95ae6e704d4cff8958eab"
Sep 17 18:45:49.552714 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:49.552694    2581 scope.go:117] "RemoveContainer" containerID="200ccdd45a2c1619d00b95834101279849889ac59bb95ae6e704d4cff8958eab"
Sep 17 18:45:49.552998 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:45:49.552980    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"200ccdd45a2c1619d00b95834101279849889ac59bb95ae6e704d4cff8958eab\": container with ID starting with 200ccdd45a2c1619d00b95834101279849889ac59bb95ae6e704d4cff8958eab not found: ID does not exist" containerID="200ccdd45a2c1619d00b95834101279849889ac59bb95ae6e704d4cff8958eab"
Sep 17 18:45:49.553072 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:49.553004    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"200ccdd45a2c1619d00b95834101279849889ac59bb95ae6e704d4cff8958eab"} err="failed to get container status \"200ccdd45a2c1619d00b95834101279849889ac59bb95ae6e704d4cff8958eab\": rpc error: code = NotFound desc = could not find container \"200ccdd45a2c1619d00b95834101279849889ac59bb95ae6e704d4cff8958eab\": container with ID starting with 200ccdd45a2c1619d00b95834101279849889ac59bb95ae6e704d4cff8958eab not found: ID does not exist"
Sep 17 18:45:49.565917 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:49.565858    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm"]
Sep 17 18:45:49.567402 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:49.567382    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdpcm"]
Sep 17 18:45:50.358562 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:50.358529    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b424313e-9217-424d-a9e4-c8b3f51d28d9" path="/var/lib/kubelet/pods/b424313e-9217-424d-a9e4-c8b3f51d28d9/volumes"
Sep 17 18:45:57.901738 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:57.901666    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm" podUID="32c04d1d-a975-48b5-9161-25dc30bc5702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 17 18:45:58.114893 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:58.114856    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl"]
Sep 17 18:45:58.115268 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:58.115253    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b424313e-9217-424d-a9e4-c8b3f51d28d9" containerName="main"
Sep 17 18:45:58.115268 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:58.115268    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b424313e-9217-424d-a9e4-c8b3f51d28d9" containerName="main"
Sep 17 18:45:58.115412 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:58.115363    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b424313e-9217-424d-a9e4-c8b3f51d28d9" containerName="main"
Sep 17 18:45:58.119782 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:58.119759    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl"
Sep 17 18:45:58.123600 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:58.123571    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 17 18:45:58.123794 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:58.123742    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 17 18:45:58.123887 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:58.123833    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-qlz97\""
Sep 17 18:45:58.124021 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:58.123664    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 17 18:45:58.124226 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:58.124197    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-rnx82\""
Sep 17 18:45:58.126212 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:58.126188    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl"]
Sep 17 18:45:58.278008 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:58.277968    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn9x7\" (UniqueName: \"kubernetes.io/projected/932b3e32-34a7-48be-994d-6bfdbc12ccea-kube-api-access-vn9x7\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl\" (UID: \"932b3e32-34a7-48be-994d-6bfdbc12ccea\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl"
Sep 17 18:45:58.278194 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:58.278024    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/932b3e32-34a7-48be-994d-6bfdbc12ccea-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl\" (UID: \"932b3e32-34a7-48be-994d-6bfdbc12ccea\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl"
Sep 17 18:45:58.379125 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:58.379085    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn9x7\" (UniqueName: \"kubernetes.io/projected/932b3e32-34a7-48be-994d-6bfdbc12ccea-kube-api-access-vn9x7\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl\" (UID: \"932b3e32-34a7-48be-994d-6bfdbc12ccea\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl"
Sep 17 18:45:58.379296 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:58.379138    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/932b3e32-34a7-48be-994d-6bfdbc12ccea-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl\" (UID: \"932b3e32-34a7-48be-994d-6bfdbc12ccea\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl"
Sep 17 18:45:58.381731 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:58.381708    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/932b3e32-34a7-48be-994d-6bfdbc12ccea-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl\" (UID: \"932b3e32-34a7-48be-994d-6bfdbc12ccea\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl"
Sep 17 18:45:58.389095 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:58.389066    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn9x7\" (UniqueName: \"kubernetes.io/projected/932b3e32-34a7-48be-994d-6bfdbc12ccea-kube-api-access-vn9x7\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl\" (UID: \"932b3e32-34a7-48be-994d-6bfdbc12ccea\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl"
Sep 17 18:45:58.432916 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:58.432866    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl"
Sep 17 18:45:58.709805 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:58.709779    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl"]
Sep 17 18:45:58.711890 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:45:58.711860    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod932b3e32_34a7_48be_994d_6bfdbc12ccea.slice/crio-1a7aa84ed30797541fc1da569553ff3eeac1208d68c5fd844b074f5254f050a9 WatchSource:0}: Error finding container 1a7aa84ed30797541fc1da569553ff3eeac1208d68c5fd844b074f5254f050a9: Status 404 returned error can't find the container with id 1a7aa84ed30797541fc1da569553ff3eeac1208d68c5fd844b074f5254f050a9
Sep 17 18:45:59.596532 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:59.596497    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl" event={"ID":"932b3e32-34a7-48be-994d-6bfdbc12ccea","Type":"ContainerStarted","Data":"41d6358fa7052c2dfe89c8066c2a5b4b30808785961871d64a2e467632139050"}
Sep 17 18:45:59.596950 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:59.596551    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl" event={"ID":"932b3e32-34a7-48be-994d-6bfdbc12ccea","Type":"ContainerStarted","Data":"1a7aa84ed30797541fc1da569553ff3eeac1208d68c5fd844b074f5254f050a9"}
Sep 17 18:45:59.596950 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:59.596612    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl"
Sep 17 18:45:59.613129 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:45:59.613067    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl" podStartSLOduration=1.61304976 podStartE2EDuration="1.61304976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:45:59.612454273 +0000 UTC m=+2947.703075935" watchObservedRunningTime="2026-09-17 18:45:59.61304976 +0000 UTC m=+2947.703671409"
Sep 17 18:46:07.911752 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:46:07.911715    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"
Sep 17 18:46:07.920187 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:46:07.920161    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"
Sep 17 18:46:30.604772 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:46:30.604737    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl"
Sep 17 18:46:44.238706 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:46:44.238675    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-9b6dc798d-57sjm_32c04d1d-a975-48b5-9161-25dc30bc5702/storage-initializer/0.log"
Sep 17 18:46:44.249645 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:46:44.249615    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-9b6dc798d-57sjm_32c04d1d-a975-48b5-9161-25dc30bc5702/main/0.log"
Sep 17 18:46:45.334540 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:46:45.334501    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"]
Sep 17 18:46:52.600862 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:46:52.600821    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/2.log"
Sep 17 18:46:52.611206 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:46:52.611181    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/2.log"
Sep 17 18:47:00.357216 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:00.357161    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm" podUID="32c04d1d-a975-48b5-9161-25dc30bc5702" containerName="main" containerID="cri-o://d8744100e43a250da2fc84a85adffb68e03585cd6604c084f7895e3943ad9c47" gracePeriod=45
Sep 17 18:47:00.902916 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:00.902861    2581 generic.go:356] "Generic (PLEG): container finished" podID="32c04d1d-a975-48b5-9161-25dc30bc5702" containerID="d8744100e43a250da2fc84a85adffb68e03585cd6604c084f7895e3943ad9c47" exitCode=0
Sep 17 18:47:00.903110 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:00.902943    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm" event={"ID":"32c04d1d-a975-48b5-9161-25dc30bc5702","Type":"ContainerDied","Data":"d8744100e43a250da2fc84a85adffb68e03585cd6604c084f7895e3943ad9c47"}
Sep 17 18:47:00.913271 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:00.913225    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm" podUID="32c04d1d-a975-48b5-9161-25dc30bc5702" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Sep 17 18:47:01.006648 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:01.006624    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"
Sep 17 18:47:01.046797 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:01.046770    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32c04d1d-a975-48b5-9161-25dc30bc5702-home\") pod \"32c04d1d-a975-48b5-9161-25dc30bc5702\" (UID: \"32c04d1d-a975-48b5-9161-25dc30bc5702\") "
Sep 17 18:47:01.046995 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:01.046815    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32c04d1d-a975-48b5-9161-25dc30bc5702-cabundle-cert\") pod \"32c04d1d-a975-48b5-9161-25dc30bc5702\" (UID: \"32c04d1d-a975-48b5-9161-25dc30bc5702\") "
Sep 17 18:47:01.046995 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:01.046851    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32c04d1d-a975-48b5-9161-25dc30bc5702-model-cache\") pod \"32c04d1d-a975-48b5-9161-25dc30bc5702\" (UID: \"32c04d1d-a975-48b5-9161-25dc30bc5702\") "
Sep 17 18:47:01.046995 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:01.046873    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqb9n\" (UniqueName: \"kubernetes.io/projected/32c04d1d-a975-48b5-9161-25dc30bc5702-kube-api-access-vqb9n\") pod \"32c04d1d-a975-48b5-9161-25dc30bc5702\" (UID: \"32c04d1d-a975-48b5-9161-25dc30bc5702\") "
Sep 17 18:47:01.046995 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:01.046913    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32c04d1d-a975-48b5-9161-25dc30bc5702-kserve-provision-location\") pod \"32c04d1d-a975-48b5-9161-25dc30bc5702\" (UID: \"32c04d1d-a975-48b5-9161-25dc30bc5702\") "
Sep 17 18:47:01.046995 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:01.046945    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32c04d1d-a975-48b5-9161-25dc30bc5702-tmp-dir\") pod \"32c04d1d-a975-48b5-9161-25dc30bc5702\" (UID: \"32c04d1d-a975-48b5-9161-25dc30bc5702\") "
Sep 17 18:47:01.047299 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:01.046999    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32c04d1d-a975-48b5-9161-25dc30bc5702-tls-certs\") pod \"32c04d1d-a975-48b5-9161-25dc30bc5702\" (UID: \"32c04d1d-a975-48b5-9161-25dc30bc5702\") "
Sep 17 18:47:01.047299 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:01.047046    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32c04d1d-a975-48b5-9161-25dc30bc5702-dshm\") pod \"32c04d1d-a975-48b5-9161-25dc30bc5702\" (UID: \"32c04d1d-a975-48b5-9161-25dc30bc5702\") "
Sep 17 18:47:01.047411 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:01.047310    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32c04d1d-a975-48b5-9161-25dc30bc5702-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "32c04d1d-a975-48b5-9161-25dc30bc5702" (UID: "32c04d1d-a975-48b5-9161-25dc30bc5702"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:47:01.047518 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:01.047497    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c04d1d-a975-48b5-9161-25dc30bc5702-model-cache" (OuterVolumeSpecName: "model-cache") pod "32c04d1d-a975-48b5-9161-25dc30bc5702" (UID: "32c04d1d-a975-48b5-9161-25dc30bc5702"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:47:01.047585 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:01.047529    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c04d1d-a975-48b5-9161-25dc30bc5702-home" (OuterVolumeSpecName: "home") pod "32c04d1d-a975-48b5-9161-25dc30bc5702" (UID: "32c04d1d-a975-48b5-9161-25dc30bc5702"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:47:01.050115 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:01.050081    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32c04d1d-a975-48b5-9161-25dc30bc5702-kube-api-access-vqb9n" (OuterVolumeSpecName: "kube-api-access-vqb9n") pod "32c04d1d-a975-48b5-9161-25dc30bc5702" (UID: "32c04d1d-a975-48b5-9161-25dc30bc5702"). InnerVolumeSpecName "kube-api-access-vqb9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:47:01.050115 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:01.050104    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c04d1d-a975-48b5-9161-25dc30bc5702-dshm" (OuterVolumeSpecName: "dshm") pod "32c04d1d-a975-48b5-9161-25dc30bc5702" (UID: "32c04d1d-a975-48b5-9161-25dc30bc5702"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:47:01.050499 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:01.050477    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32c04d1d-a975-48b5-9161-25dc30bc5702-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "32c04d1d-a975-48b5-9161-25dc30bc5702" (UID: "32c04d1d-a975-48b5-9161-25dc30bc5702"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:47:01.066113 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:01.066071    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c04d1d-a975-48b5-9161-25dc30bc5702-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "32c04d1d-a975-48b5-9161-25dc30bc5702" (UID: "32c04d1d-a975-48b5-9161-25dc30bc5702"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:47:01.072655 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:01.072626    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c04d1d-a975-48b5-9161-25dc30bc5702-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "32c04d1d-a975-48b5-9161-25dc30bc5702" (UID: "32c04d1d-a975-48b5-9161-25dc30bc5702"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:47:01.148301 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:01.148267    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32c04d1d-a975-48b5-9161-25dc30bc5702-home\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:47:01.148301 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:01.148295    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32c04d1d-a975-48b5-9161-25dc30bc5702-cabundle-cert\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:47:01.148301 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:01.148304    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32c04d1d-a975-48b5-9161-25dc30bc5702-model-cache\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:47:01.148301 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:01.148312    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqb9n\" (UniqueName: \"kubernetes.io/projected/32c04d1d-a975-48b5-9161-25dc30bc5702-kube-api-access-vqb9n\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:47:01.148564 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:01.148320    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32c04d1d-a975-48b5-9161-25dc30bc5702-kserve-provision-location\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:47:01.148564 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:01.148329    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32c04d1d-a975-48b5-9161-25dc30bc5702-tmp-dir\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:47:01.148564 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:01.148338    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32c04d1d-a975-48b5-9161-25dc30bc5702-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:47:01.148564 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:01.148345    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32c04d1d-a975-48b5-9161-25dc30bc5702-dshm\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:47:01.909335 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:01.909288    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm" event={"ID":"32c04d1d-a975-48b5-9161-25dc30bc5702","Type":"ContainerDied","Data":"a7b2e95c933c07891be5c2ef2a5b7163b0f234f03c8e71c8441eb13ae9effaf9"}
Sep 17 18:47:01.909739 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:01.909388    2581 scope.go:117] "RemoveContainer" containerID="d8744100e43a250da2fc84a85adffb68e03585cd6604c084f7895e3943ad9c47"
Sep 17 18:47:01.909739 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:01.909305    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"
Sep 17 18:47:01.918704 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:01.918676    2581 scope.go:117] "RemoveContainer" containerID="67b0ec7b951fa67d18e7df728bbe6606de472ff8e880a902710e215913fdd2e3"
Sep 17 18:47:01.932468 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:01.932439    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"]
Sep 17 18:47:01.938183 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:01.938157    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-9b6dc798d-57sjm"]
Sep 17 18:47:02.358203 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:02.358167    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32c04d1d-a975-48b5-9161-25dc30bc5702" path="/var/lib/kubelet/pods/32c04d1d-a975-48b5-9161-25dc30bc5702/volumes"
Sep 17 18:47:12.975428 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:12.975392    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"]
Sep 17 18:47:12.976279 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:12.976251    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32c04d1d-a975-48b5-9161-25dc30bc5702" containerName="storage-initializer"
Sep 17 18:47:12.976279 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:12.976280    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="32c04d1d-a975-48b5-9161-25dc30bc5702" containerName="storage-initializer"
Sep 17 18:47:12.976487 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:12.976290    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32c04d1d-a975-48b5-9161-25dc30bc5702" containerName="main"
Sep 17 18:47:12.976487 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:12.976298    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="32c04d1d-a975-48b5-9161-25dc30bc5702" containerName="main"
Sep 17 18:47:12.976487 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:12.976403    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="32c04d1d-a975-48b5-9161-25dc30bc5702" containerName="main"
Sep 17 18:47:12.979572 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:12.979551    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"
Sep 17 18:47:12.982451 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:12.982429    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Sep 17 18:47:12.982582 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:12.982554    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Sep 17 18:47:12.982702 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:12.982582    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-6f5qr\""
Sep 17 18:47:12.982830 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:12.982812    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:47:12.983028 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:12.983008    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Sep 17 18:47:12.983742 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:12.983726    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Sep 17 18:47:12.989490 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:12.989471    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"]
Sep 17 18:47:13.061158 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:13.061128    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b85e2c22-5b1a-4854-95c0-4897c47a7863-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-774f47469f-fzmh6\" (UID: \"b85e2c22-5b1a-4854-95c0-4897c47a7863\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"
Sep 17 18:47:13.061322 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:13.061175    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b85e2c22-5b1a-4854-95c0-4897c47a7863-model-cache\") pod \"lora-multiple-adapters-test-kserve-774f47469f-fzmh6\" (UID: \"b85e2c22-5b1a-4854-95c0-4897c47a7863\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"
Sep 17 18:47:13.061322 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:13.061207    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcgl6\" (UniqueName: \"kubernetes.io/projected/b85e2c22-5b1a-4854-95c0-4897c47a7863-kube-api-access-jcgl6\") pod \"lora-multiple-adapters-test-kserve-774f47469f-fzmh6\" (UID: \"b85e2c22-5b1a-4854-95c0-4897c47a7863\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"
Sep 17 18:47:13.061322 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:13.061228    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b85e2c22-5b1a-4854-95c0-4897c47a7863-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-774f47469f-fzmh6\" (UID: \"b85e2c22-5b1a-4854-95c0-4897c47a7863\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"
Sep 17 18:47:13.061322 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:13.061245    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b85e2c22-5b1a-4854-95c0-4897c47a7863-dshm\") pod \"lora-multiple-adapters-test-kserve-774f47469f-fzmh6\" (UID: \"b85e2c22-5b1a-4854-95c0-4897c47a7863\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"
Sep 17 18:47:13.061322 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:13.061266    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b85e2c22-5b1a-4854-95c0-4897c47a7863-tls-certs\") pod \"lora-multiple-adapters-test-kserve-774f47469f-fzmh6\" (UID: \"b85e2c22-5b1a-4854-95c0-4897c47a7863\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"
Sep 17 18:47:13.061558 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:13.061348    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b85e2c22-5b1a-4854-95c0-4897c47a7863-home\") pod \"lora-multiple-adapters-test-kserve-774f47469f-fzmh6\" (UID: \"b85e2c22-5b1a-4854-95c0-4897c47a7863\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"
Sep 17 18:47:13.061558 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:13.061407    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b85e2c22-5b1a-4854-95c0-4897c47a7863-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-774f47469f-fzmh6\" (UID: \"b85e2c22-5b1a-4854-95c0-4897c47a7863\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"
Sep 17 18:47:13.162600 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:13.162547    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b85e2c22-5b1a-4854-95c0-4897c47a7863-model-cache\") pod \"lora-multiple-adapters-test-kserve-774f47469f-fzmh6\" (UID: \"b85e2c22-5b1a-4854-95c0-4897c47a7863\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"
Sep 17 18:47:13.162600 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:13.162615    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcgl6\" (UniqueName: \"kubernetes.io/projected/b85e2c22-5b1a-4854-95c0-4897c47a7863-kube-api-access-jcgl6\") pod \"lora-multiple-adapters-test-kserve-774f47469f-fzmh6\" (UID: \"b85e2c22-5b1a-4854-95c0-4897c47a7863\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"
Sep 17 18:47:13.162867 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:13.162636    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b85e2c22-5b1a-4854-95c0-4897c47a7863-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-774f47469f-fzmh6\" (UID: \"b85e2c22-5b1a-4854-95c0-4897c47a7863\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"
Sep 17 18:47:13.162867 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:13.162655    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b85e2c22-5b1a-4854-95c0-4897c47a7863-dshm\") pod \"lora-multiple-adapters-test-kserve-774f47469f-fzmh6\" (UID: \"b85e2c22-5b1a-4854-95c0-4897c47a7863\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"
Sep 17 18:47:13.162867 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:13.162681    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b85e2c22-5b1a-4854-95c0-4897c47a7863-tls-certs\") pod \"lora-multiple-adapters-test-kserve-774f47469f-fzmh6\" (UID: \"b85e2c22-5b1a-4854-95c0-4897c47a7863\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"
Sep 17 18:47:13.162867 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:13.162709    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b85e2c22-5b1a-4854-95c0-4897c47a7863-home\") pod \"lora-multiple-adapters-test-kserve-774f47469f-fzmh6\" (UID: \"b85e2c22-5b1a-4854-95c0-4897c47a7863\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"
Sep 17 18:47:13.162867 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:13.162740    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b85e2c22-5b1a-4854-95c0-4897c47a7863-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-774f47469f-fzmh6\" (UID: \"b85e2c22-5b1a-4854-95c0-4897c47a7863\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"
Sep 17 18:47:13.162867 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:13.162780    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b85e2c22-5b1a-4854-95c0-4897c47a7863-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-774f47469f-fzmh6\" (UID: \"b85e2c22-5b1a-4854-95c0-4897c47a7863\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"
Sep 17 18:47:13.163206 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:13.163103    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b85e2c22-5b1a-4854-95c0-4897c47a7863-model-cache\") pod \"lora-multiple-adapters-test-kserve-774f47469f-fzmh6\" (UID: \"b85e2c22-5b1a-4854-95c0-4897c47a7863\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"
Sep 17 18:47:13.163206 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:13.163128    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b85e2c22-5b1a-4854-95c0-4897c47a7863-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-774f47469f-fzmh6\" (UID: \"b85e2c22-5b1a-4854-95c0-4897c47a7863\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"
Sep 17 18:47:13.163206 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:13.163150    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b85e2c22-5b1a-4854-95c0-4897c47a7863-home\") pod \"lora-multiple-adapters-test-kserve-774f47469f-fzmh6\" (UID: \"b85e2c22-5b1a-4854-95c0-4897c47a7863\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"
Sep 17 18:47:13.163338 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:13.163314    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b85e2c22-5b1a-4854-95c0-4897c47a7863-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-774f47469f-fzmh6\" (UID: \"b85e2c22-5b1a-4854-95c0-4897c47a7863\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"
Sep 17 18:47:13.163601 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:13.163577    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b85e2c22-5b1a-4854-95c0-4897c47a7863-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-774f47469f-fzmh6\" (UID: \"b85e2c22-5b1a-4854-95c0-4897c47a7863\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"
Sep 17 18:47:13.165054 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:13.165034    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b85e2c22-5b1a-4854-95c0-4897c47a7863-dshm\") pod \"lora-multiple-adapters-test-kserve-774f47469f-fzmh6\" (UID: \"b85e2c22-5b1a-4854-95c0-4897c47a7863\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"
Sep 17 18:47:13.165424 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:13.165407    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b85e2c22-5b1a-4854-95c0-4897c47a7863-tls-certs\") pod \"lora-multiple-adapters-test-kserve-774f47469f-fzmh6\" (UID: \"b85e2c22-5b1a-4854-95c0-4897c47a7863\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"
Sep 17 18:47:13.170537 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:13.170510    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcgl6\" (UniqueName: \"kubernetes.io/projected/b85e2c22-5b1a-4854-95c0-4897c47a7863-kube-api-access-jcgl6\") pod \"lora-multiple-adapters-test-kserve-774f47469f-fzmh6\" (UID: \"b85e2c22-5b1a-4854-95c0-4897c47a7863\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"
Sep 17 18:47:13.290543 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:13.290446    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"
Sep 17 18:47:13.574286 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:13.574191    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"]
Sep 17 18:47:13.578177 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:47:13.578139    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb85e2c22_5b1a_4854_95c0_4897c47a7863.slice/crio-a66b3b56a65d81288ae891bf84c52650b0bfb0b62be0d2caca0515061d9876dd WatchSource:0}: Error finding container a66b3b56a65d81288ae891bf84c52650b0bfb0b62be0d2caca0515061d9876dd: Status 404 returned error can't find the container with id a66b3b56a65d81288ae891bf84c52650b0bfb0b62be0d2caca0515061d9876dd
Sep 17 18:47:13.972212 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:13.972175    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6" event={"ID":"b85e2c22-5b1a-4854-95c0-4897c47a7863","Type":"ContainerStarted","Data":"ba12cba5549e7cbc3e9e4d04aa904706b2b9b300be3f97ad7ad6d1684bbb6fd4"}
Sep 17 18:47:13.972400 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:13.972247    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6" event={"ID":"b85e2c22-5b1a-4854-95c0-4897c47a7863","Type":"ContainerStarted","Data":"a66b3b56a65d81288ae891bf84c52650b0bfb0b62be0d2caca0515061d9876dd"}
Sep 17 18:47:16.988547 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:16.988510    2581 generic.go:356] "Generic (PLEG): container finished" podID="b85e2c22-5b1a-4854-95c0-4897c47a7863" containerID="ba12cba5549e7cbc3e9e4d04aa904706b2b9b300be3f97ad7ad6d1684bbb6fd4" exitCode=0
Sep 17 18:47:16.988937 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:16.988581    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6" event={"ID":"b85e2c22-5b1a-4854-95c0-4897c47a7863","Type":"ContainerDied","Data":"ba12cba5549e7cbc3e9e4d04aa904706b2b9b300be3f97ad7ad6d1684bbb6fd4"}
Sep 17 18:47:17.996249 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:17.996198    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6" event={"ID":"b85e2c22-5b1a-4854-95c0-4897c47a7863","Type":"ContainerStarted","Data":"a798435387dab3c5a39f15827d7b33b05f391d8e291bafe5c8fc24a691266b20"}
Sep 17 18:47:18.019186 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:18.019110    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6" podStartSLOduration=3.052161909 podStartE2EDuration="6.019094681s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.966932772s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:47:18.018846047 +0000 UTC m=+3026.109467739" watchObservedRunningTime="2026-09-17 18:47:18.019094681 +0000 UTC m=+3026.109716330"
Sep 17 18:47:18.291066 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:18.290970    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"
Sep 17 18:47:23.291283 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:23.291237    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"
Sep 17 18:47:23.293140 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:23.293111    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6" podUID="b85e2c22-5b1a-4854-95c0-4897c47a7863" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 17 18:47:33.290987 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:33.290939    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6" podUID="b85e2c22-5b1a-4854-95c0-4897c47a7863" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 17 18:47:43.292048 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:43.291988    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6" podUID="b85e2c22-5b1a-4854-95c0-4897c47a7863" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 17 18:47:53.291423 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:47:53.291369    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6" podUID="b85e2c22-5b1a-4854-95c0-4897c47a7863" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 17 18:48:03.291062 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:48:03.291015    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6" podUID="b85e2c22-5b1a-4854-95c0-4897c47a7863" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 17 18:48:13.291586 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:48:13.291534    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6" podUID="b85e2c22-5b1a-4854-95c0-4897c47a7863" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 17 18:48:23.291066 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:48:23.291012    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6" podUID="b85e2c22-5b1a-4854-95c0-4897c47a7863" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 17 18:48:33.301815 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:48:33.301777    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"
Sep 17 18:48:33.309754 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:48:33.309723    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"
Sep 17 18:48:58.122800 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:48:58.122760    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl"]
Sep 17 18:49:00.604436 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:00.604398    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl" podUID="932b3e32-34a7-48be-994d-6bfdbc12ccea" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:49:01.056640 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:01.056609    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-774f47469f-fzmh6_b85e2c22-5b1a-4854-95c0-4897c47a7863/storage-initializer/0.log"
Sep 17 18:49:01.066124 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:01.066099    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-774f47469f-fzmh6_b85e2c22-5b1a-4854-95c0-4897c47a7863/main/0.log"
Sep 17 18:49:02.106232 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:02.106199    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"]
Sep 17 18:49:10.603718 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:10.603677    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl" podUID="932b3e32-34a7-48be-994d-6bfdbc12ccea" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:49:13.158760 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:13.158714    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl" podUID="932b3e32-34a7-48be-994d-6bfdbc12ccea" containerName="main" containerID="cri-o://41d6358fa7052c2dfe89c8066c2a5b4b30808785961871d64a2e467632139050" gracePeriod=45
Sep 17 18:49:17.131152 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:17.131063    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6" podUID="b85e2c22-5b1a-4854-95c0-4897c47a7863" containerName="main" containerID="cri-o://a798435387dab3c5a39f15827d7b33b05f391d8e291bafe5c8fc24a691266b20" gracePeriod=45
Sep 17 18:49:17.302974 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:17.302932    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6" podUID="b85e2c22-5b1a-4854-95c0-4897c47a7863" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Sep 17 18:49:17.779122 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:17.779091    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"
Sep 17 18:49:17.832233 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:17.832134    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b85e2c22-5b1a-4854-95c0-4897c47a7863-home\") pod \"b85e2c22-5b1a-4854-95c0-4897c47a7863\" (UID: \"b85e2c22-5b1a-4854-95c0-4897c47a7863\") "
Sep 17 18:49:17.832233 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:17.832185    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b85e2c22-5b1a-4854-95c0-4897c47a7863-cabundle-cert\") pod \"b85e2c22-5b1a-4854-95c0-4897c47a7863\" (UID: \"b85e2c22-5b1a-4854-95c0-4897c47a7863\") "
Sep 17 18:49:17.832233 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:17.832216    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b85e2c22-5b1a-4854-95c0-4897c47a7863-dshm\") pod \"b85e2c22-5b1a-4854-95c0-4897c47a7863\" (UID: \"b85e2c22-5b1a-4854-95c0-4897c47a7863\") "
Sep 17 18:49:17.832527 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:17.832248    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcgl6\" (UniqueName: \"kubernetes.io/projected/b85e2c22-5b1a-4854-95c0-4897c47a7863-kube-api-access-jcgl6\") pod \"b85e2c22-5b1a-4854-95c0-4897c47a7863\" (UID: \"b85e2c22-5b1a-4854-95c0-4897c47a7863\") "
Sep 17 18:49:17.832527 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:17.832265    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b85e2c22-5b1a-4854-95c0-4897c47a7863-kserve-provision-location\") pod \"b85e2c22-5b1a-4854-95c0-4897c47a7863\" (UID: \"b85e2c22-5b1a-4854-95c0-4897c47a7863\") "
Sep 17 18:49:17.832527 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:17.832300    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b85e2c22-5b1a-4854-95c0-4897c47a7863-model-cache\") pod \"b85e2c22-5b1a-4854-95c0-4897c47a7863\" (UID: \"b85e2c22-5b1a-4854-95c0-4897c47a7863\") "
Sep 17 18:49:17.832527 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:17.832342    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b85e2c22-5b1a-4854-95c0-4897c47a7863-tls-certs\") pod \"b85e2c22-5b1a-4854-95c0-4897c47a7863\" (UID: \"b85e2c22-5b1a-4854-95c0-4897c47a7863\") "
Sep 17 18:49:17.832527 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:17.832372    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b85e2c22-5b1a-4854-95c0-4897c47a7863-tmp-dir\") pod \"b85e2c22-5b1a-4854-95c0-4897c47a7863\" (UID: \"b85e2c22-5b1a-4854-95c0-4897c47a7863\") "
Sep 17 18:49:17.832768 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:17.832660    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b85e2c22-5b1a-4854-95c0-4897c47a7863-home" (OuterVolumeSpecName: "home") pod "b85e2c22-5b1a-4854-95c0-4897c47a7863" (UID: "b85e2c22-5b1a-4854-95c0-4897c47a7863"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:49:17.833059 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:17.833032    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b85e2c22-5b1a-4854-95c0-4897c47a7863-model-cache" (OuterVolumeSpecName: "model-cache") pod "b85e2c22-5b1a-4854-95c0-4897c47a7863" (UID: "b85e2c22-5b1a-4854-95c0-4897c47a7863"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:49:17.833175 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:17.833071    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b85e2c22-5b1a-4854-95c0-4897c47a7863-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b85e2c22-5b1a-4854-95c0-4897c47a7863" (UID: "b85e2c22-5b1a-4854-95c0-4897c47a7863"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:49:17.835299 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:17.835267    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b85e2c22-5b1a-4854-95c0-4897c47a7863-dshm" (OuterVolumeSpecName: "dshm") pod "b85e2c22-5b1a-4854-95c0-4897c47a7863" (UID: "b85e2c22-5b1a-4854-95c0-4897c47a7863"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:49:17.835410 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:17.835340    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b85e2c22-5b1a-4854-95c0-4897c47a7863-kube-api-access-jcgl6" (OuterVolumeSpecName: "kube-api-access-jcgl6") pod "b85e2c22-5b1a-4854-95c0-4897c47a7863" (UID: "b85e2c22-5b1a-4854-95c0-4897c47a7863"). InnerVolumeSpecName "kube-api-access-jcgl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:49:17.835478 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:17.835414    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b85e2c22-5b1a-4854-95c0-4897c47a7863-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b85e2c22-5b1a-4854-95c0-4897c47a7863" (UID: "b85e2c22-5b1a-4854-95c0-4897c47a7863"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:49:17.844958 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:17.844927    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b85e2c22-5b1a-4854-95c0-4897c47a7863-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b85e2c22-5b1a-4854-95c0-4897c47a7863" (UID: "b85e2c22-5b1a-4854-95c0-4897c47a7863"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:49:17.859085 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:17.859057    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b85e2c22-5b1a-4854-95c0-4897c47a7863-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b85e2c22-5b1a-4854-95c0-4897c47a7863" (UID: "b85e2c22-5b1a-4854-95c0-4897c47a7863"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:49:17.933219 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:17.933178    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b85e2c22-5b1a-4854-95c0-4897c47a7863-home\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:49:17.933219 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:17.933213    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b85e2c22-5b1a-4854-95c0-4897c47a7863-cabundle-cert\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:49:17.933219 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:17.933223    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b85e2c22-5b1a-4854-95c0-4897c47a7863-dshm\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:49:17.933453 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:17.933234    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcgl6\" (UniqueName: \"kubernetes.io/projected/b85e2c22-5b1a-4854-95c0-4897c47a7863-kube-api-access-jcgl6\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:49:17.933453 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:17.933243    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b85e2c22-5b1a-4854-95c0-4897c47a7863-kserve-provision-location\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:49:17.933453 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:17.933253    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b85e2c22-5b1a-4854-95c0-4897c47a7863-model-cache\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:49:17.933453 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:17.933265    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b85e2c22-5b1a-4854-95c0-4897c47a7863-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:49:17.933453 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:17.933275    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b85e2c22-5b1a-4854-95c0-4897c47a7863-tmp-dir\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:49:18.584267 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:18.584231    2581 generic.go:356] "Generic (PLEG): container finished" podID="b85e2c22-5b1a-4854-95c0-4897c47a7863" containerID="a798435387dab3c5a39f15827d7b33b05f391d8e291bafe5c8fc24a691266b20" exitCode=0
Sep 17 18:49:18.584740 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:18.584288    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"
Sep 17 18:49:18.584740 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:18.584291    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6" event={"ID":"b85e2c22-5b1a-4854-95c0-4897c47a7863","Type":"ContainerDied","Data":"a798435387dab3c5a39f15827d7b33b05f391d8e291bafe5c8fc24a691266b20"}
Sep 17 18:49:18.584740 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:18.584406    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6" event={"ID":"b85e2c22-5b1a-4854-95c0-4897c47a7863","Type":"ContainerDied","Data":"a66b3b56a65d81288ae891bf84c52650b0bfb0b62be0d2caca0515061d9876dd"}
Sep 17 18:49:18.584740 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:18.584431    2581 scope.go:117] "RemoveContainer" containerID="a798435387dab3c5a39f15827d7b33b05f391d8e291bafe5c8fc24a691266b20"
Sep 17 18:49:18.593079 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:18.593059    2581 scope.go:117] "RemoveContainer" containerID="ba12cba5549e7cbc3e9e4d04aa904706b2b9b300be3f97ad7ad6d1684bbb6fd4"
Sep 17 18:49:18.601040 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:18.601019    2581 scope.go:117] "RemoveContainer" containerID="a798435387dab3c5a39f15827d7b33b05f391d8e291bafe5c8fc24a691266b20"
Sep 17 18:49:18.601406 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:49:18.601384    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a798435387dab3c5a39f15827d7b33b05f391d8e291bafe5c8fc24a691266b20\": container with ID starting with a798435387dab3c5a39f15827d7b33b05f391d8e291bafe5c8fc24a691266b20 not found: ID does not exist" containerID="a798435387dab3c5a39f15827d7b33b05f391d8e291bafe5c8fc24a691266b20"
Sep 17 18:49:18.601508 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:18.601418    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a798435387dab3c5a39f15827d7b33b05f391d8e291bafe5c8fc24a691266b20"} err="failed to get container status \"a798435387dab3c5a39f15827d7b33b05f391d8e291bafe5c8fc24a691266b20\": rpc error: code = NotFound desc = could not find container \"a798435387dab3c5a39f15827d7b33b05f391d8e291bafe5c8fc24a691266b20\": container with ID starting with a798435387dab3c5a39f15827d7b33b05f391d8e291bafe5c8fc24a691266b20 not found: ID does not exist"
Sep 17 18:49:18.601508 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:18.601437    2581 scope.go:117] "RemoveContainer" containerID="ba12cba5549e7cbc3e9e4d04aa904706b2b9b300be3f97ad7ad6d1684bbb6fd4"
Sep 17 18:49:18.601770 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:49:18.601739    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba12cba5549e7cbc3e9e4d04aa904706b2b9b300be3f97ad7ad6d1684bbb6fd4\": container with ID starting with ba12cba5549e7cbc3e9e4d04aa904706b2b9b300be3f97ad7ad6d1684bbb6fd4 not found: ID does not exist" containerID="ba12cba5549e7cbc3e9e4d04aa904706b2b9b300be3f97ad7ad6d1684bbb6fd4"
Sep 17 18:49:18.601886 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:18.601779    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba12cba5549e7cbc3e9e4d04aa904706b2b9b300be3f97ad7ad6d1684bbb6fd4"} err="failed to get container status \"ba12cba5549e7cbc3e9e4d04aa904706b2b9b300be3f97ad7ad6d1684bbb6fd4\": rpc error: code = NotFound desc = could not find container \"ba12cba5549e7cbc3e9e4d04aa904706b2b9b300be3f97ad7ad6d1684bbb6fd4\": container with ID starting with ba12cba5549e7cbc3e9e4d04aa904706b2b9b300be3f97ad7ad6d1684bbb6fd4 not found: ID does not exist"
Sep 17 18:49:18.603145 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:18.603124    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"]
Sep 17 18:49:18.606724 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:18.606703    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-774f47469f-fzmh6"]
Sep 17 18:49:20.358662 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:20.358627    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b85e2c22-5b1a-4854-95c0-4897c47a7863" path="/var/lib/kubelet/pods/b85e2c22-5b1a-4854-95c0-4897c47a7863/volumes"
Sep 17 18:49:20.604002 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:20.603963    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl" podUID="932b3e32-34a7-48be-994d-6bfdbc12ccea" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:49:20.604194 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:20.604092    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl"
Sep 17 18:49:30.604627 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:30.604585    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl" podUID="932b3e32-34a7-48be-994d-6bfdbc12ccea" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:49:40.604570 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:40.604526    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl" podUID="932b3e32-34a7-48be-994d-6bfdbc12ccea" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:49:43.404501 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:43.404474    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl"
Sep 17 18:49:43.565884 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:43.565849    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vn9x7\" (UniqueName: \"kubernetes.io/projected/932b3e32-34a7-48be-994d-6bfdbc12ccea-kube-api-access-vn9x7\") pod \"932b3e32-34a7-48be-994d-6bfdbc12ccea\" (UID: \"932b3e32-34a7-48be-994d-6bfdbc12ccea\") "
Sep 17 18:49:43.566078 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:43.566048    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/932b3e32-34a7-48be-994d-6bfdbc12ccea-tls-certs\") pod \"932b3e32-34a7-48be-994d-6bfdbc12ccea\" (UID: \"932b3e32-34a7-48be-994d-6bfdbc12ccea\") "
Sep 17 18:49:43.568124 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:43.568088    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/932b3e32-34a7-48be-994d-6bfdbc12ccea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "932b3e32-34a7-48be-994d-6bfdbc12ccea" (UID: "932b3e32-34a7-48be-994d-6bfdbc12ccea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:49:43.568249 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:43.568152    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/932b3e32-34a7-48be-994d-6bfdbc12ccea-kube-api-access-vn9x7" (OuterVolumeSpecName: "kube-api-access-vn9x7") pod "932b3e32-34a7-48be-994d-6bfdbc12ccea" (UID: "932b3e32-34a7-48be-994d-6bfdbc12ccea"). InnerVolumeSpecName "kube-api-access-vn9x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:49:43.666730 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:43.666692    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn9x7\" (UniqueName: \"kubernetes.io/projected/932b3e32-34a7-48be-994d-6bfdbc12ccea-kube-api-access-vn9x7\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:49:43.666730 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:43.666726    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/932b3e32-34a7-48be-994d-6bfdbc12ccea-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:49:43.702064 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:43.702030    2581 generic.go:356] "Generic (PLEG): container finished" podID="932b3e32-34a7-48be-994d-6bfdbc12ccea" containerID="41d6358fa7052c2dfe89c8066c2a5b4b30808785961871d64a2e467632139050" exitCode=0
Sep 17 18:49:43.702198 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:43.702073    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl" event={"ID":"932b3e32-34a7-48be-994d-6bfdbc12ccea","Type":"ContainerDied","Data":"41d6358fa7052c2dfe89c8066c2a5b4b30808785961871d64a2e467632139050"}
Sep 17 18:49:43.702198 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:43.702094    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl"
Sep 17 18:49:43.702198 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:43.702154    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl" event={"ID":"932b3e32-34a7-48be-994d-6bfdbc12ccea","Type":"ContainerDied","Data":"1a7aa84ed30797541fc1da569553ff3eeac1208d68c5fd844b074f5254f050a9"}
Sep 17 18:49:43.702198 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:43.702185    2581 scope.go:117] "RemoveContainer" containerID="41d6358fa7052c2dfe89c8066c2a5b4b30808785961871d64a2e467632139050"
Sep 17 18:49:43.711088 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:43.711071    2581 scope.go:117] "RemoveContainer" containerID="41d6358fa7052c2dfe89c8066c2a5b4b30808785961871d64a2e467632139050"
Sep 17 18:49:43.711345 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:49:43.711326    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41d6358fa7052c2dfe89c8066c2a5b4b30808785961871d64a2e467632139050\": container with ID starting with 41d6358fa7052c2dfe89c8066c2a5b4b30808785961871d64a2e467632139050 not found: ID does not exist" containerID="41d6358fa7052c2dfe89c8066c2a5b4b30808785961871d64a2e467632139050"
Sep 17 18:49:43.711409 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:43.711350    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d6358fa7052c2dfe89c8066c2a5b4b30808785961871d64a2e467632139050"} err="failed to get container status \"41d6358fa7052c2dfe89c8066c2a5b4b30808785961871d64a2e467632139050\": rpc error: code = NotFound desc = could not find container \"41d6358fa7052c2dfe89c8066c2a5b4b30808785961871d64a2e467632139050\": container with ID starting with 41d6358fa7052c2dfe89c8066c2a5b4b30808785961871d64a2e467632139050 not found: ID does not exist"
Sep 17 18:49:43.725812 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:43.725788    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl"]
Sep 17 18:49:43.729419 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:43.729396    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c484nqzl"]
Sep 17 18:49:44.357509 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:44.357476    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="932b3e32-34a7-48be-994d-6bfdbc12ccea" path="/var/lib/kubelet/pods/932b3e32-34a7-48be-994d-6bfdbc12ccea/volumes"
Sep 17 18:49:48.465057 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.465016    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"]
Sep 17 18:49:48.466233 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.466197    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="932b3e32-34a7-48be-994d-6bfdbc12ccea" containerName="main"
Sep 17 18:49:48.466400 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.466387    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="932b3e32-34a7-48be-994d-6bfdbc12ccea" containerName="main"
Sep 17 18:49:48.466551 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.466538    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b85e2c22-5b1a-4854-95c0-4897c47a7863" containerName="main"
Sep 17 18:49:48.466667 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.466655    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b85e2c22-5b1a-4854-95c0-4897c47a7863" containerName="main"
Sep 17 18:49:48.466782 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.466771    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b85e2c22-5b1a-4854-95c0-4897c47a7863" containerName="storage-initializer"
Sep 17 18:49:48.466920 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.466878    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b85e2c22-5b1a-4854-95c0-4897c47a7863" containerName="storage-initializer"
Sep 17 18:49:48.467112 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.467094    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="932b3e32-34a7-48be-994d-6bfdbc12ccea" containerName="main"
Sep 17 18:49:48.467221 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.467207    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b85e2c22-5b1a-4854-95c0-4897c47a7863" containerName="main"
Sep 17 18:49:48.471641 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.471619    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:49:48.474795 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.474774    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Sep 17 18:49:48.474972 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.474778    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-mc9rn\""
Sep 17 18:49:48.475135 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.474885    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 17 18:49:48.475491 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.475473    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 17 18:49:48.483447 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.483422    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"]
Sep 17 18:49:48.621287 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.621238    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-77d9dd58dd-cx7hh\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:49:48.621596 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.621572    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-workload-socket\") pod \"router-gateway-2-openshift-default-77d9dd58dd-cx7hh\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:49:48.621769 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.621752    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-workload-certs\") pod \"router-gateway-2-openshift-default-77d9dd58dd-cx7hh\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:49:48.621992 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.621955    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-istio-data\") pod \"router-gateway-2-openshift-default-77d9dd58dd-cx7hh\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:49:48.622085 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.622038    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-credential-socket\") pod \"router-gateway-2-openshift-default-77d9dd58dd-cx7hh\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:49:48.622085 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.622071    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-istio-podinfo\") pod \"router-gateway-2-openshift-default-77d9dd58dd-cx7hh\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:49:48.622209 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.622106    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8kqc\" (UniqueName: \"kubernetes.io/projected/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-kube-api-access-t8kqc\") pod \"router-gateway-2-openshift-default-77d9dd58dd-cx7hh\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:49:48.622209 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.622135    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-istio-envoy\") pod \"router-gateway-2-openshift-default-77d9dd58dd-cx7hh\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:49:48.622209 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.622173    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-istio-token\") pod \"router-gateway-2-openshift-default-77d9dd58dd-cx7hh\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:49:48.723168 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.723052    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-workload-socket\") pod \"router-gateway-2-openshift-default-77d9dd58dd-cx7hh\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:49:48.723575 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.723550    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-workload-certs\") pod \"router-gateway-2-openshift-default-77d9dd58dd-cx7hh\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:49:48.723737 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.723721    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-istio-data\") pod \"router-gateway-2-openshift-default-77d9dd58dd-cx7hh\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:49:48.723918 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.723839    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-workload-certs\") pod \"router-gateway-2-openshift-default-77d9dd58dd-cx7hh\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:49:48.723918 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.723868    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-credential-socket\") pod \"router-gateway-2-openshift-default-77d9dd58dd-cx7hh\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:49:48.723918 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.723784    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-workload-socket\") pod \"router-gateway-2-openshift-default-77d9dd58dd-cx7hh\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:49:48.724171 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.723921    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-istio-podinfo\") pod \"router-gateway-2-openshift-default-77d9dd58dd-cx7hh\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:49:48.724171 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.723960    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8kqc\" (UniqueName: \"kubernetes.io/projected/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-kube-api-access-t8kqc\") pod \"router-gateway-2-openshift-default-77d9dd58dd-cx7hh\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:49:48.724171 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.723989    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-istio-envoy\") pod \"router-gateway-2-openshift-default-77d9dd58dd-cx7hh\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:49:48.724171 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.724028    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-istio-token\") pod \"router-gateway-2-openshift-default-77d9dd58dd-cx7hh\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:49:48.724171 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.724109    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-77d9dd58dd-cx7hh\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:49:48.724171 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.724105    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-istio-data\") pod \"router-gateway-2-openshift-default-77d9dd58dd-cx7hh\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:49:48.725136 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.724872    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-credential-socket\") pod \"router-gateway-2-openshift-default-77d9dd58dd-cx7hh\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:49:48.725594 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.725566    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-77d9dd58dd-cx7hh\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:49:48.730174 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.730149    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-istio-envoy\") pod \"router-gateway-2-openshift-default-77d9dd58dd-cx7hh\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:49:48.731810 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.731765    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-istio-podinfo\") pod \"router-gateway-2-openshift-default-77d9dd58dd-cx7hh\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:49:48.738471 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.738442    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-istio-token\") pod \"router-gateway-2-openshift-default-77d9dd58dd-cx7hh\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:49:48.738701 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.738678    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8kqc\" (UniqueName: \"kubernetes.io/projected/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-kube-api-access-t8kqc\") pod \"router-gateway-2-openshift-default-77d9dd58dd-cx7hh\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:49:48.785644 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:48.785605    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:49:49.089808 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:49.089753    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"]
Sep 17 18:49:49.092663 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:49:49.092631    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebe139c6_bbeb_47bd_8966_4e3629fc6ce5.slice/crio-5baa8a6ea1c105e7cd80f8b98f87c697d6c0995798aa1792c5258638e28e4562 WatchSource:0}: Error finding container 5baa8a6ea1c105e7cd80f8b98f87c697d6c0995798aa1792c5258638e28e4562: Status 404 returned error can't find the container with id 5baa8a6ea1c105e7cd80f8b98f87c697d6c0995798aa1792c5258638e28e4562
Sep 17 18:49:49.095016 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:49.094984    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:49:49.095062 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:49.095051    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:49:49.095102 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:49.095087    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:49:49.739465 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:49.739420    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh" event={"ID":"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5","Type":"ContainerStarted","Data":"d7691e3f3cf0bbf94f740b09acaa762a3822fc2e540f64aebb8f5b3b8f19f2ee"}
Sep 17 18:49:49.739826 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:49.739515    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh" event={"ID":"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5","Type":"ContainerStarted","Data":"5baa8a6ea1c105e7cd80f8b98f87c697d6c0995798aa1792c5258638e28e4562"}
Sep 17 18:49:49.761893 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:49.761832    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh" podStartSLOduration=1.761815943 podStartE2EDuration="1.761815943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:49:49.760651762 +0000 UTC m=+3177.851273440" watchObservedRunningTime="2026-09-17 18:49:49.761815943 +0000 UTC m=+3177.852437592"
Sep 17 18:49:49.786735 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:49.786697    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:49:50.790687 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:50.790650    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:49:51.749968 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:51.749936    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:49:51.750961 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:49:51.750940    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:50:00.796729 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:00.796694    2581 generic.go:356] "Generic (PLEG): container finished" podID="ebe139c6-bbeb-47bd-8966-4e3629fc6ce5" containerID="d7691e3f3cf0bbf94f740b09acaa762a3822fc2e540f64aebb8f5b3b8f19f2ee" exitCode=0
Sep 17 18:50:00.796729 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:00.796737    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh" event={"ID":"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5","Type":"ContainerDied","Data":"d7691e3f3cf0bbf94f740b09acaa762a3822fc2e540f64aebb8f5b3b8f19f2ee"}
Sep 17 18:50:00.797351 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:00.797260    2581 scope.go:117] "RemoveContainer" containerID="d7691e3f3cf0bbf94f740b09acaa762a3822fc2e540f64aebb8f5b3b8f19f2ee"
Sep 17 18:50:00.798296 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:00.798262    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:50:00.798370 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:00.798353    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:50:00.798417 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:00.798403    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 18:50:01.803077 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:01.803041    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh" event={"ID":"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5","Type":"ContainerStarted","Data":"916265bb35139ba4a5f4ddb7703c8175c31a5e0ccbe600ffb9b826bfc788bea5"}
Sep 17 18:50:02.786261 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:02.786224    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:50:02.787408 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:02.787368    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh" podUID="ebe139c6-bbeb-47bd-8966-4e3629fc6ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 17 18:50:03.786348 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:03.786300    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh" podUID="ebe139c6-bbeb-47bd-8966-4e3629fc6ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 17 18:50:04.786188 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:04.786142    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh" podUID="ebe139c6-bbeb-47bd-8966-4e3629fc6ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 17 18:50:05.786072 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:05.786033    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh" podUID="ebe139c6-bbeb-47bd-8966-4e3629fc6ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 17 18:50:06.750162 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:06.750119    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:50:06.786529 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:06.786493    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh" podUID="ebe139c6-bbeb-47bd-8966-4e3629fc6ce5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 17 18:50:07.790513 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:07.790478    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:50:07.791264 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:07.791244    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:50:45.587515 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:45.587435    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"]
Sep 17 18:50:45.590288 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:45.590265    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:50:45.592830 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:45.592805    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 17 18:50:45.592982 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:45.592894    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-sp9wg\""
Sep 17 18:50:45.592982 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:45.592928    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-jdxkb\""
Sep 17 18:50:45.593124 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:45.593068    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:50:45.593209 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:45.593191    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 17 18:50:45.606233 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:45.606204    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"]
Sep 17 18:50:45.664294 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:45.664260    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-tls-certs\") pod \"router-with-refs-pd-test-kserve-766657c59f-ncqb8\" (UID: \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:50:45.664456 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:45.664303    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-dshm\") pod \"router-with-refs-pd-test-kserve-766657c59f-ncqb8\" (UID: \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:50:45.664456 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:45.664323    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-tmp-dir\") pod \"router-with-refs-pd-test-kserve-766657c59f-ncqb8\" (UID: \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:50:45.664549 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:45.664457    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-766657c59f-ncqb8\" (UID: \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:50:45.664549 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:45.664492    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-766657c59f-ncqb8\" (UID: \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:50:45.664549 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:45.664516    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-model-cache\") pod \"router-with-refs-pd-test-kserve-766657c59f-ncqb8\" (UID: \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:50:45.664662 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:45.664554    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfc9q\" (UniqueName: \"kubernetes.io/projected/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-kube-api-access-jfc9q\") pod \"router-with-refs-pd-test-kserve-766657c59f-ncqb8\" (UID: \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:50:45.664662 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:45.664599    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-home\") pod \"router-with-refs-pd-test-kserve-766657c59f-ncqb8\" (UID: \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:50:45.765630 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:45.765583    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-766657c59f-ncqb8\" (UID: \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:50:45.765841 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:45.765637    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-766657c59f-ncqb8\" (UID: \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:50:45.765841 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:45.765672    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-model-cache\") pod \"router-with-refs-pd-test-kserve-766657c59f-ncqb8\" (UID: \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:50:45.765841 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:45.765694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfc9q\" (UniqueName: \"kubernetes.io/projected/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-kube-api-access-jfc9q\") pod \"router-with-refs-pd-test-kserve-766657c59f-ncqb8\" (UID: \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:50:45.765841 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:45.765732    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-home\") pod \"router-with-refs-pd-test-kserve-766657c59f-ncqb8\" (UID: \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:50:45.765841 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:45.765764    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-tls-certs\") pod \"router-with-refs-pd-test-kserve-766657c59f-ncqb8\" (UID: \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:50:45.765841 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:45.765805    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-dshm\") pod \"router-with-refs-pd-test-kserve-766657c59f-ncqb8\" (UID: \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:50:45.765841 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:45.765828    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-tmp-dir\") pod \"router-with-refs-pd-test-kserve-766657c59f-ncqb8\" (UID: \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:50:45.766482 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:45.766221    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-home\") pod \"router-with-refs-pd-test-kserve-766657c59f-ncqb8\" (UID: \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:50:45.766482 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:45.766265    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-766657c59f-ncqb8\" (UID: \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:50:45.766599 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:45.766490    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-766657c59f-ncqb8\" (UID: \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:50:45.766599 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:45.766520    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-model-cache\") pod \"router-with-refs-pd-test-kserve-766657c59f-ncqb8\" (UID: \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:50:45.766599 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:45.766540    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-tmp-dir\") pod \"router-with-refs-pd-test-kserve-766657c59f-ncqb8\" (UID: \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:50:45.768165 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:45.768136    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-dshm\") pod \"router-with-refs-pd-test-kserve-766657c59f-ncqb8\" (UID: \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:50:45.768455 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:45.768434    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-tls-certs\") pod \"router-with-refs-pd-test-kserve-766657c59f-ncqb8\" (UID: \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:50:45.774882 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:45.774856    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfc9q\" (UniqueName: \"kubernetes.io/projected/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-kube-api-access-jfc9q\") pod \"router-with-refs-pd-test-kserve-766657c59f-ncqb8\" (UID: \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:50:45.901327 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:45.901218    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:50:46.043365 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:46.043323    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d"]
Sep 17 18:50:46.046194 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:46.046173    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d"
Sep 17 18:50:46.048752 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:46.048727    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-97ghq\""
Sep 17 18:50:46.054696 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:46.054665    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d"]
Sep 17 18:50:46.169985 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:46.169882    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d\" (UID: \"89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d"
Sep 17 18:50:46.170145 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:46.170041    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49lhg\" (UniqueName: \"kubernetes.io/projected/89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88-kube-api-access-49lhg\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d\" (UID: \"89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d"
Sep 17 18:50:46.179347 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:46.179323    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"]
Sep 17 18:50:46.181032 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:50:46.181004    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a9f59e6_7e28_4d19_92a8_dbf193a9e0ef.slice/crio-77a6f2a7fead0ea5318320f55e846c2971e56645296fe4d95eeb4b73a2d42480 WatchSource:0}: Error finding container 77a6f2a7fead0ea5318320f55e846c2971e56645296fe4d95eeb4b73a2d42480: Status 404 returned error can't find the container with id 77a6f2a7fead0ea5318320f55e846c2971e56645296fe4d95eeb4b73a2d42480
Sep 17 18:50:46.271186 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:46.271150    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49lhg\" (UniqueName: \"kubernetes.io/projected/89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88-kube-api-access-49lhg\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d\" (UID: \"89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d"
Sep 17 18:50:46.271375 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:46.271209    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d\" (UID: \"89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d"
Sep 17 18:50:46.273700 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:46.273675    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d\" (UID: \"89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d"
Sep 17 18:50:46.278769 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:46.278746    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49lhg\" (UniqueName: \"kubernetes.io/projected/89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88-kube-api-access-49lhg\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d\" (UID: \"89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d"
Sep 17 18:50:46.356598 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:46.356567    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d"
Sep 17 18:50:46.630646 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:46.630619    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d"]
Sep 17 18:50:46.633262 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:50:46.633218    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89ce1f86_a1b5_4a3e_9f8b_8d8707e8fa88.slice/crio-5451a3c8cbccbc32b4db376990bc6eed0f8b5aa22a7529fbae9e6f54bf02191e WatchSource:0}: Error finding container 5451a3c8cbccbc32b4db376990bc6eed0f8b5aa22a7529fbae9e6f54bf02191e: Status 404 returned error can't find the container with id 5451a3c8cbccbc32b4db376990bc6eed0f8b5aa22a7529fbae9e6f54bf02191e
Sep 17 18:50:47.022643 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:47.022608    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8" event={"ID":"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef","Type":"ContainerStarted","Data":"d1ac0c0333a12a5158f3abc177e4664fa020a91300736f282bcfbd9e1e43731c"}
Sep 17 18:50:47.022861 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:47.022666    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8" event={"ID":"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef","Type":"ContainerStarted","Data":"77a6f2a7fead0ea5318320f55e846c2971e56645296fe4d95eeb4b73a2d42480"}
Sep 17 18:50:47.022861 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:47.022715    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:50:47.024410 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:47.024383    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d" event={"ID":"89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88","Type":"ContainerStarted","Data":"e3a4f708b25fdd166b2d3ae53662319d53514274ffcb7df781aa9e39d5afe08b"}
Sep 17 18:50:47.024527 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:47.024431    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d" event={"ID":"89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88","Type":"ContainerStarted","Data":"5451a3c8cbccbc32b4db376990bc6eed0f8b5aa22a7529fbae9e6f54bf02191e"}
Sep 17 18:50:47.024527 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:47.024474    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d"
Sep 17 18:50:47.105149 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:47.105085    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d" podStartSLOduration=1.105064213 podStartE2EDuration="1.105064213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:50:47.103728315 +0000 UTC m=+3235.194349975" watchObservedRunningTime="2026-09-17 18:50:47.105064213 +0000 UTC m=+3235.195685865"
Sep 17 18:50:48.035616 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:48.035568    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8" event={"ID":"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef","Type":"ContainerStarted","Data":"7338f3416b96781348b136f929ba4ff7d06b89c5c4e51e161ddd6a619ab4c205"}
Sep 17 18:50:50.047019 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:50.046982    2581 generic.go:356] "Generic (PLEG): container finished" podID="0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef" containerID="7338f3416b96781348b136f929ba4ff7d06b89c5c4e51e161ddd6a619ab4c205" exitCode=0
Sep 17 18:50:50.047413 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:50.047050    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8" event={"ID":"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef","Type":"ContainerDied","Data":"7338f3416b96781348b136f929ba4ff7d06b89c5c4e51e161ddd6a619ab4c205"}
Sep 17 18:50:51.054301 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:51.054265    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8" event={"ID":"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef","Type":"ContainerStarted","Data":"3c99feca731b0d05c88facf083aafb11c84e082576748eeb1633c08124f4a09e"}
Sep 17 18:50:55.901845 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:55.901805    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:50:55.902261 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:55.901875    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:50:55.903129 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:50:55.903098    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8" podUID="0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Sep 17 18:51:05.902026 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:51:05.901972    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8" podUID="0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Sep 17 18:51:05.931207 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:51:05.931175    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:51:05.962982 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:51:05.962890    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8" podStartSLOduration=18.569739984 podStartE2EDuration="20.962869836s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.393129852s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:50:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:50:51.07937233 +0000 UTC m=+3239.169993979" watchObservedRunningTime="2026-09-17 18:51:05.962869836 +0000 UTC m=+3254.053491487"
Sep 17 18:51:15.901861 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:51:15.901816    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8" podUID="0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Sep 17 18:51:18.038153 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:51:18.038123    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d"
Sep 17 18:51:25.902564 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:51:25.902515    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8" podUID="0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Sep 17 18:51:35.901988 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:51:35.901933    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8" podUID="0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Sep 17 18:51:45.902717 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:51:45.902654    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8" podUID="0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Sep 17 18:51:52.638749 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:51:52.638708    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/2.log"
Sep 17 18:51:52.653074 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:51:52.653047    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/2.log"
Sep 17 18:51:55.901639 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:51:55.901587    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8" podUID="0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Sep 17 18:52:05.901926 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:52:05.901845    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8" podUID="0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Sep 17 18:52:15.902372 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:52:15.902257    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8" podUID="0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Sep 17 18:52:25.902484 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:52:25.902429    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8" podUID="0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Sep 17 18:52:35.902310 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:52:35.902257    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8" podUID="0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Sep 17 18:52:45.911691 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:52:45.911656    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:52:45.923785 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:52:45.923756    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:53:07.240753 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:07.240691    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d"]
Sep 17 18:53:07.262816 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:07.262782    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"]
Sep 17 18:53:08.038001 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:08.037958    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d" podUID="89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:53:18.038060 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:18.038021    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d" podUID="89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:53:22.261269 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:22.261224    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d" podUID="89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88" containerName="main" containerID="cri-o://e3a4f708b25fdd166b2d3ae53662319d53514274ffcb7df781aa9e39d5afe08b" gracePeriod=45
Sep 17 18:53:22.285264 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:22.285188    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8" podUID="0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef" containerName="main" containerID="cri-o://3c99feca731b0d05c88facf083aafb11c84e082576748eeb1633c08124f4a09e" gracePeriod=45
Sep 17 18:53:22.901775 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:22.901721    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8" podUID="0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef" containerName="llm-d-routing-sidecar" containerID="cri-o://d1ac0c0333a12a5158f3abc177e4664fa020a91300736f282bcfbd9e1e43731c" gracePeriod=45
Sep 17 18:53:23.170522 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.170494    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:53:23.295083 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.295026    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-cabundle-cert\") pod \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\" (UID: \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\") "
Sep 17 18:53:23.295570 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.295104    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-dshm\") pod \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\" (UID: \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\") "
Sep 17 18:53:23.295570 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.295136    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfc9q\" (UniqueName: \"kubernetes.io/projected/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-kube-api-access-jfc9q\") pod \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\" (UID: \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\") "
Sep 17 18:53:23.295570 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.295199    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-tmp-dir\") pod \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\" (UID: \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\") "
Sep 17 18:53:23.295570 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.295224    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-home\") pod \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\" (UID: \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\") "
Sep 17 18:53:23.295570 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.295254    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-kserve-provision-location\") pod \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\" (UID: \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\") "
Sep 17 18:53:23.295570 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.295312    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-tls-certs\") pod \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\" (UID: \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\") "
Sep 17 18:53:23.295570 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.295390    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-model-cache\") pod \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\" (UID: \"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef\") "
Sep 17 18:53:23.296008 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.295491    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef" (UID: "0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:53:23.296008 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.295731    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-cabundle-cert\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:53:23.296008 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.295790    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-home" (OuterVolumeSpecName: "home") pod "0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef" (UID: "0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:53:23.296008 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.295832    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-model-cache" (OuterVolumeSpecName: "model-cache") pod "0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef" (UID: "0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:53:23.298062 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.298033    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-kube-api-access-jfc9q" (OuterVolumeSpecName: "kube-api-access-jfc9q") pod "0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef" (UID: "0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef"). InnerVolumeSpecName "kube-api-access-jfc9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:53:23.298062 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.298045    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-dshm" (OuterVolumeSpecName: "dshm") pod "0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef" (UID: "0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:53:23.298233 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.298144    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef" (UID: "0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:53:23.310542 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.310509    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef" (UID: "0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:53:23.321631 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.321544    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef" (UID: "0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:53:23.396704 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.396659    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-tmp-dir\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:53:23.396704 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.396698    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-home\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:53:23.396704 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.396711    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-kserve-provision-location\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:53:23.396988 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.396720    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:53:23.396988 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.396731    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-model-cache\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:53:23.396988 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.396739    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-dshm\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:53:23.396988 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.396748    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfc9q\" (UniqueName: \"kubernetes.io/projected/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef-kube-api-access-jfc9q\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:53:23.816066 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.816030    2581 generic.go:356] "Generic (PLEG): container finished" podID="0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef" containerID="3c99feca731b0d05c88facf083aafb11c84e082576748eeb1633c08124f4a09e" exitCode=0
Sep 17 18:53:23.816066 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.816059    2581 generic.go:356] "Generic (PLEG): container finished" podID="0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef" containerID="d1ac0c0333a12a5158f3abc177e4664fa020a91300736f282bcfbd9e1e43731c" exitCode=0
Sep 17 18:53:23.816300 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.816085    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8" event={"ID":"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef","Type":"ContainerDied","Data":"3c99feca731b0d05c88facf083aafb11c84e082576748eeb1633c08124f4a09e"}
Sep 17 18:53:23.816300 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.816104    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"
Sep 17 18:53:23.816300 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.816161    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8" event={"ID":"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef","Type":"ContainerDied","Data":"d1ac0c0333a12a5158f3abc177e4664fa020a91300736f282bcfbd9e1e43731c"}
Sep 17 18:53:23.816300 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.816179    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8" event={"ID":"0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef","Type":"ContainerDied","Data":"77a6f2a7fead0ea5318320f55e846c2971e56645296fe4d95eeb4b73a2d42480"}
Sep 17 18:53:23.816300 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.816204    2581 scope.go:117] "RemoveContainer" containerID="3c99feca731b0d05c88facf083aafb11c84e082576748eeb1633c08124f4a09e"
Sep 17 18:53:23.826087 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.826062    2581 scope.go:117] "RemoveContainer" containerID="7338f3416b96781348b136f929ba4ff7d06b89c5c4e51e161ddd6a619ab4c205"
Sep 17 18:53:23.834040 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.834020    2581 scope.go:117] "RemoveContainer" containerID="d1ac0c0333a12a5158f3abc177e4664fa020a91300736f282bcfbd9e1e43731c"
Sep 17 18:53:23.841764 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.841743    2581 scope.go:117] "RemoveContainer" containerID="3c99feca731b0d05c88facf083aafb11c84e082576748eeb1633c08124f4a09e"
Sep 17 18:53:23.842087 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:53:23.842065    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c99feca731b0d05c88facf083aafb11c84e082576748eeb1633c08124f4a09e\": container with ID starting with 3c99feca731b0d05c88facf083aafb11c84e082576748eeb1633c08124f4a09e not found: ID does not exist" containerID="3c99feca731b0d05c88facf083aafb11c84e082576748eeb1633c08124f4a09e"
Sep 17 18:53:23.842139 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.842097    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c99feca731b0d05c88facf083aafb11c84e082576748eeb1633c08124f4a09e"} err="failed to get container status \"3c99feca731b0d05c88facf083aafb11c84e082576748eeb1633c08124f4a09e\": rpc error: code = NotFound desc = could not find container \"3c99feca731b0d05c88facf083aafb11c84e082576748eeb1633c08124f4a09e\": container with ID starting with 3c99feca731b0d05c88facf083aafb11c84e082576748eeb1633c08124f4a09e not found: ID does not exist"
Sep 17 18:53:23.842139 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.842115    2581 scope.go:117] "RemoveContainer" containerID="7338f3416b96781348b136f929ba4ff7d06b89c5c4e51e161ddd6a619ab4c205"
Sep 17 18:53:23.842369 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:53:23.842350    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7338f3416b96781348b136f929ba4ff7d06b89c5c4e51e161ddd6a619ab4c205\": container with ID starting with 7338f3416b96781348b136f929ba4ff7d06b89c5c4e51e161ddd6a619ab4c205 not found: ID does not exist" containerID="7338f3416b96781348b136f929ba4ff7d06b89c5c4e51e161ddd6a619ab4c205"
Sep 17 18:53:23.842415 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.842376    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7338f3416b96781348b136f929ba4ff7d06b89c5c4e51e161ddd6a619ab4c205"} err="failed to get container status \"7338f3416b96781348b136f929ba4ff7d06b89c5c4e51e161ddd6a619ab4c205\": rpc error: code = NotFound desc = could not find container \"7338f3416b96781348b136f929ba4ff7d06b89c5c4e51e161ddd6a619ab4c205\": container with ID starting with 7338f3416b96781348b136f929ba4ff7d06b89c5c4e51e161ddd6a619ab4c205 not found: ID does not exist"
Sep 17 18:53:23.842415 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.842392    2581 scope.go:117] "RemoveContainer" containerID="d1ac0c0333a12a5158f3abc177e4664fa020a91300736f282bcfbd9e1e43731c"
Sep 17 18:53:23.842639 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:53:23.842623    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1ac0c0333a12a5158f3abc177e4664fa020a91300736f282bcfbd9e1e43731c\": container with ID starting with d1ac0c0333a12a5158f3abc177e4664fa020a91300736f282bcfbd9e1e43731c not found: ID does not exist" containerID="d1ac0c0333a12a5158f3abc177e4664fa020a91300736f282bcfbd9e1e43731c"
Sep 17 18:53:23.842704 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.842647    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1ac0c0333a12a5158f3abc177e4664fa020a91300736f282bcfbd9e1e43731c"} err="failed to get container status \"d1ac0c0333a12a5158f3abc177e4664fa020a91300736f282bcfbd9e1e43731c\": rpc error: code = NotFound desc = could not find container \"d1ac0c0333a12a5158f3abc177e4664fa020a91300736f282bcfbd9e1e43731c\": container with ID starting with d1ac0c0333a12a5158f3abc177e4664fa020a91300736f282bcfbd9e1e43731c not found: ID does not exist"
Sep 17 18:53:23.842704 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.842667    2581 scope.go:117] "RemoveContainer" containerID="3c99feca731b0d05c88facf083aafb11c84e082576748eeb1633c08124f4a09e"
Sep 17 18:53:23.842924 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.842892    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c99feca731b0d05c88facf083aafb11c84e082576748eeb1633c08124f4a09e"} err="failed to get container status \"3c99feca731b0d05c88facf083aafb11c84e082576748eeb1633c08124f4a09e\": rpc error: code = NotFound desc = could not find container \"3c99feca731b0d05c88facf083aafb11c84e082576748eeb1633c08124f4a09e\": container with ID starting with 3c99feca731b0d05c88facf083aafb11c84e082576748eeb1633c08124f4a09e not found: ID does not exist"
Sep 17 18:53:23.842924 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.842924    2581 scope.go:117] "RemoveContainer" containerID="7338f3416b96781348b136f929ba4ff7d06b89c5c4e51e161ddd6a619ab4c205"
Sep 17 18:53:23.843135 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.843113    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7338f3416b96781348b136f929ba4ff7d06b89c5c4e51e161ddd6a619ab4c205"} err="failed to get container status \"7338f3416b96781348b136f929ba4ff7d06b89c5c4e51e161ddd6a619ab4c205\": rpc error: code = NotFound desc = could not find container \"7338f3416b96781348b136f929ba4ff7d06b89c5c4e51e161ddd6a619ab4c205\": container with ID starting with 7338f3416b96781348b136f929ba4ff7d06b89c5c4e51e161ddd6a619ab4c205 not found: ID does not exist"
Sep 17 18:53:23.843179 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.843135    2581 scope.go:117] "RemoveContainer" containerID="d1ac0c0333a12a5158f3abc177e4664fa020a91300736f282bcfbd9e1e43731c"
Sep 17 18:53:23.843335 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.843313    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1ac0c0333a12a5158f3abc177e4664fa020a91300736f282bcfbd9e1e43731c"} err="failed to get container status \"d1ac0c0333a12a5158f3abc177e4664fa020a91300736f282bcfbd9e1e43731c\": rpc error: code = NotFound desc = could not find container \"d1ac0c0333a12a5158f3abc177e4664fa020a91300736f282bcfbd9e1e43731c\": container with ID starting with d1ac0c0333a12a5158f3abc177e4664fa020a91300736f282bcfbd9e1e43731c not found: ID does not exist"
Sep 17 18:53:23.846837 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.846811    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"]
Sep 17 18:53:23.853279 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:23.853255    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-766657c59f-ncqb8"]
Sep 17 18:53:24.358444 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:24.358411    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef" path="/var/lib/kubelet/pods/0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef/volumes"
Sep 17 18:53:28.037361 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:28.037323    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d" podUID="89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:53:28.037733 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:28.037442    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d"
Sep 17 18:53:38.038374 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:38.038327    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d" podUID="89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:53:48.038099 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:48.038006    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d" podUID="89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:53:52.508865 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:52.508837    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d"
Sep 17 18:53:52.570883 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:52.570789    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49lhg\" (UniqueName: \"kubernetes.io/projected/89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88-kube-api-access-49lhg\") pod \"89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88\" (UID: \"89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88\") "
Sep 17 18:53:52.570883 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:52.570833    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88-tls-certs\") pod \"89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88\" (UID: \"89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88\") "
Sep 17 18:53:52.573114 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:52.573074    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88" (UID: "89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:53:52.573228 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:52.573145    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88-kube-api-access-49lhg" (OuterVolumeSpecName: "kube-api-access-49lhg") pod "89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88" (UID: "89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88"). InnerVolumeSpecName "kube-api-access-49lhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:53:52.671760 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:52.671717    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49lhg\" (UniqueName: \"kubernetes.io/projected/89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88-kube-api-access-49lhg\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:53:52.671760 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:52.671751    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:53:52.955016 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:52.954981    2581 generic.go:356] "Generic (PLEG): container finished" podID="89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88" containerID="e3a4f708b25fdd166b2d3ae53662319d53514274ffcb7df781aa9e39d5afe08b" exitCode=0
Sep 17 18:53:52.955193 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:52.955024    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d"
Sep 17 18:53:52.955193 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:52.955026    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d" event={"ID":"89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88","Type":"ContainerDied","Data":"e3a4f708b25fdd166b2d3ae53662319d53514274ffcb7df781aa9e39d5afe08b"}
Sep 17 18:53:52.955193 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:52.955156    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d" event={"ID":"89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88","Type":"ContainerDied","Data":"5451a3c8cbccbc32b4db376990bc6eed0f8b5aa22a7529fbae9e6f54bf02191e"}
Sep 17 18:53:52.955193 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:52.955182    2581 scope.go:117] "RemoveContainer" containerID="e3a4f708b25fdd166b2d3ae53662319d53514274ffcb7df781aa9e39d5afe08b"
Sep 17 18:53:52.965360 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:52.965336    2581 scope.go:117] "RemoveContainer" containerID="e3a4f708b25fdd166b2d3ae53662319d53514274ffcb7df781aa9e39d5afe08b"
Sep 17 18:53:52.965699 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:53:52.965678    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3a4f708b25fdd166b2d3ae53662319d53514274ffcb7df781aa9e39d5afe08b\": container with ID starting with e3a4f708b25fdd166b2d3ae53662319d53514274ffcb7df781aa9e39d5afe08b not found: ID does not exist" containerID="e3a4f708b25fdd166b2d3ae53662319d53514274ffcb7df781aa9e39d5afe08b"
Sep 17 18:53:52.965754 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:52.965709    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3a4f708b25fdd166b2d3ae53662319d53514274ffcb7df781aa9e39d5afe08b"} err="failed to get container status \"e3a4f708b25fdd166b2d3ae53662319d53514274ffcb7df781aa9e39d5afe08b\": rpc error: code = NotFound desc = could not find container \"e3a4f708b25fdd166b2d3ae53662319d53514274ffcb7df781aa9e39d5afe08b\": container with ID starting with e3a4f708b25fdd166b2d3ae53662319d53514274ffcb7df781aa9e39d5afe08b not found: ID does not exist"
Sep 17 18:53:52.978095 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:52.978063    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d"]
Sep 17 18:53:52.983645 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:52.983613    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b454n45d"]
Sep 17 18:53:54.358517 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:53:54.358481    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88" path="/var/lib/kubelet/pods/89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88/volumes"
Sep 17 18:54:00.588630 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.588596    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"]
Sep 17 18:54:00.589029 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.589002    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef" containerName="storage-initializer"
Sep 17 18:54:00.589029 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.589020    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef" containerName="storage-initializer"
Sep 17 18:54:00.589154 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.589048    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef" containerName="llm-d-routing-sidecar"
Sep 17 18:54:00.589154 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.589058    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef" containerName="llm-d-routing-sidecar"
Sep 17 18:54:00.589154 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.589085    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88" containerName="main"
Sep 17 18:54:00.589154 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.589093    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88" containerName="main"
Sep 17 18:54:00.589154 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.589104    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef" containerName="main"
Sep 17 18:54:00.589154 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.589113    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef" containerName="main"
Sep 17 18:54:00.589458 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.589185    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef" containerName="llm-d-routing-sidecar"
Sep 17 18:54:00.589458 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.589200    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a9f59e6-7e28-4d19-92a8-dbf193a9e0ef" containerName="main"
Sep 17 18:54:00.589458 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.589213    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="89ce1f86-a1b5-4a3e-9f8b-8d8707e8fa88" containerName="main"
Sep 17 18:54:00.594065 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.594042    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"
Sep 17 18:54:00.596813 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.596790    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Sep 17 18:54:00.596943 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.596796    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:54:00.597086 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.597063    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Sep 17 18:54:00.597206 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.597079    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-qlhcj\""
Sep 17 18:54:00.597206 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.597140    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Sep 17 18:54:00.597307 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.597232    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Sep 17 18:54:00.604152 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.604131    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"]
Sep 17 18:54:00.642800 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.642770    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57699c84-5d0c-476b-93f8-e2dcb48524da-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r\" (UID: \"57699c84-5d0c-476b-93f8-e2dcb48524da\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"
Sep 17 18:54:00.642960 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.642813    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57699c84-5d0c-476b-93f8-e2dcb48524da-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r\" (UID: \"57699c84-5d0c-476b-93f8-e2dcb48524da\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"
Sep 17 18:54:00.642960 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.642831    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57699c84-5d0c-476b-93f8-e2dcb48524da-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r\" (UID: \"57699c84-5d0c-476b-93f8-e2dcb48524da\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"
Sep 17 18:54:00.642960 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.642863    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67rtx\" (UniqueName: \"kubernetes.io/projected/57699c84-5d0c-476b-93f8-e2dcb48524da-kube-api-access-67rtx\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r\" (UID: \"57699c84-5d0c-476b-93f8-e2dcb48524da\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"
Sep 17 18:54:00.642960 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.642928    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57699c84-5d0c-476b-93f8-e2dcb48524da-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r\" (UID: \"57699c84-5d0c-476b-93f8-e2dcb48524da\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"
Sep 17 18:54:00.643110 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.642965    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57699c84-5d0c-476b-93f8-e2dcb48524da-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r\" (UID: \"57699c84-5d0c-476b-93f8-e2dcb48524da\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"
Sep 17 18:54:00.643110 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.642988    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57699c84-5d0c-476b-93f8-e2dcb48524da-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r\" (UID: \"57699c84-5d0c-476b-93f8-e2dcb48524da\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"
Sep 17 18:54:00.643110 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.643016    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57699c84-5d0c-476b-93f8-e2dcb48524da-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r\" (UID: \"57699c84-5d0c-476b-93f8-e2dcb48524da\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"
Sep 17 18:54:00.743455 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.743418    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57699c84-5d0c-476b-93f8-e2dcb48524da-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r\" (UID: \"57699c84-5d0c-476b-93f8-e2dcb48524da\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"
Sep 17 18:54:00.743455 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.743454    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57699c84-5d0c-476b-93f8-e2dcb48524da-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r\" (UID: \"57699c84-5d0c-476b-93f8-e2dcb48524da\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"
Sep 17 18:54:00.743781 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.743475    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67rtx\" (UniqueName: \"kubernetes.io/projected/57699c84-5d0c-476b-93f8-e2dcb48524da-kube-api-access-67rtx\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r\" (UID: \"57699c84-5d0c-476b-93f8-e2dcb48524da\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"
Sep 17 18:54:00.743781 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.743518    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57699c84-5d0c-476b-93f8-e2dcb48524da-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r\" (UID: \"57699c84-5d0c-476b-93f8-e2dcb48524da\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"
Sep 17 18:54:00.743781 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.743553    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57699c84-5d0c-476b-93f8-e2dcb48524da-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r\" (UID: \"57699c84-5d0c-476b-93f8-e2dcb48524da\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"
Sep 17 18:54:00.743781 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.743583    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57699c84-5d0c-476b-93f8-e2dcb48524da-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r\" (UID: \"57699c84-5d0c-476b-93f8-e2dcb48524da\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"
Sep 17 18:54:00.743781 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.743616    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57699c84-5d0c-476b-93f8-e2dcb48524da-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r\" (UID: \"57699c84-5d0c-476b-93f8-e2dcb48524da\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"
Sep 17 18:54:00.743781 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.743712    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57699c84-5d0c-476b-93f8-e2dcb48524da-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r\" (UID: \"57699c84-5d0c-476b-93f8-e2dcb48524da\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"
Sep 17 18:54:00.744146 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.743858    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57699c84-5d0c-476b-93f8-e2dcb48524da-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r\" (UID: \"57699c84-5d0c-476b-93f8-e2dcb48524da\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"
Sep 17 18:54:00.744146 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.743962    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57699c84-5d0c-476b-93f8-e2dcb48524da-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r\" (UID: \"57699c84-5d0c-476b-93f8-e2dcb48524da\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"
Sep 17 18:54:00.744146 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.744065    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57699c84-5d0c-476b-93f8-e2dcb48524da-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r\" (UID: \"57699c84-5d0c-476b-93f8-e2dcb48524da\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"
Sep 17 18:54:00.744345 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.744186    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57699c84-5d0c-476b-93f8-e2dcb48524da-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r\" (UID: \"57699c84-5d0c-476b-93f8-e2dcb48524da\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"
Sep 17 18:54:00.744387 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.744352    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57699c84-5d0c-476b-93f8-e2dcb48524da-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r\" (UID: \"57699c84-5d0c-476b-93f8-e2dcb48524da\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"
Sep 17 18:54:00.745807 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.745777    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57699c84-5d0c-476b-93f8-e2dcb48524da-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r\" (UID: \"57699c84-5d0c-476b-93f8-e2dcb48524da\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"
Sep 17 18:54:00.746164 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.746147    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57699c84-5d0c-476b-93f8-e2dcb48524da-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r\" (UID: \"57699c84-5d0c-476b-93f8-e2dcb48524da\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"
Sep 17 18:54:00.752600 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.752580    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67rtx\" (UniqueName: \"kubernetes.io/projected/57699c84-5d0c-476b-93f8-e2dcb48524da-kube-api-access-67rtx\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r\" (UID: \"57699c84-5d0c-476b-93f8-e2dcb48524da\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"
Sep 17 18:54:00.889858 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.889763    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"]
Sep 17 18:54:00.890029 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.889977    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh" podUID="ebe139c6-bbeb-47bd-8966-4e3629fc6ce5" containerName="istio-proxy" containerID="cri-o://916265bb35139ba4a5f4ddb7703c8175c31a5e0ccbe600ffb9b826bfc788bea5" gracePeriod=30
Sep 17 18:54:00.905090 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:00.905064    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"
Sep 17 18:54:01.189151 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:01.189122    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"]
Sep 17 18:54:01.192509 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:54:01.192478    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57699c84_5d0c_476b_93f8_e2dcb48524da.slice/crio-3c7c5c3878a78e5e5effb52b6d8fdbf1fbf78b34a2dfc94cc57e1a757aeec1a9 WatchSource:0}: Error finding container 3c7c5c3878a78e5e5effb52b6d8fdbf1fbf78b34a2dfc94cc57e1a757aeec1a9: Status 404 returned error can't find the container with id 3c7c5c3878a78e5e5effb52b6d8fdbf1fbf78b34a2dfc94cc57e1a757aeec1a9
Sep 17 18:54:02.006978 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:02.006934    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r" event={"ID":"57699c84-5d0c-476b-93f8-e2dcb48524da","Type":"ContainerStarted","Data":"0d21d4fb0cd8903902b429a22d8fc1318099fc41e7193a20d9e9672424e12c9e"}
Sep 17 18:54:02.007399 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:02.007007    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r" event={"ID":"57699c84-5d0c-476b-93f8-e2dcb48524da","Type":"ContainerStarted","Data":"3c7c5c3878a78e5e5effb52b6d8fdbf1fbf78b34a2dfc94cc57e1a757aeec1a9"}
Sep 17 18:54:04.020393 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:04.020348    2581 generic.go:356] "Generic (PLEG): container finished" podID="57699c84-5d0c-476b-93f8-e2dcb48524da" containerID="0d21d4fb0cd8903902b429a22d8fc1318099fc41e7193a20d9e9672424e12c9e" exitCode=0
Sep 17 18:54:04.020783 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:04.020423    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r" event={"ID":"57699c84-5d0c-476b-93f8-e2dcb48524da","Type":"ContainerDied","Data":"0d21d4fb0cd8903902b429a22d8fc1318099fc41e7193a20d9e9672424e12c9e"}
Sep 17 18:54:05.028042 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:05.027980    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r" event={"ID":"57699c84-5d0c-476b-93f8-e2dcb48524da","Type":"ContainerStarted","Data":"f14759ad1c6de2c3fedbd19de1a1a59b6700f535c6da6018cad950e025ae0f20"}
Sep 17 18:54:05.053158 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:05.053089    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r" podStartSLOduration=2.6645122900000002 podStartE2EDuration="5.053069529s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.388557239s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:54:05.051414521 +0000 UTC m=+3433.142036172" watchObservedRunningTime="2026-09-17 18:54:05.053069529 +0000 UTC m=+3433.143691179"
Sep 17 18:54:05.905871 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:05.905829    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"
Sep 17 18:54:06.037360 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:06.037317    2581 generic.go:356] "Generic (PLEG): container finished" podID="ebe139c6-bbeb-47bd-8966-4e3629fc6ce5" containerID="916265bb35139ba4a5f4ddb7703c8175c31a5e0ccbe600ffb9b826bfc788bea5" exitCode=0
Sep 17 18:54:06.037746 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:06.037387    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh" event={"ID":"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5","Type":"ContainerDied","Data":"916265bb35139ba4a5f4ddb7703c8175c31a5e0ccbe600ffb9b826bfc788bea5"}
Sep 17 18:54:06.037746 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:06.037449    2581 scope.go:117] "RemoveContainer" containerID="d7691e3f3cf0bbf94f740b09acaa762a3822fc2e540f64aebb8f5b3b8f19f2ee"
Sep 17 18:54:06.151026 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:06.150948    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:54:06.201031 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:06.200996    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-istio-token\") pod \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") "
Sep 17 18:54:06.201229 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:06.201076    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-istio-podinfo\") pod \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") "
Sep 17 18:54:06.201229 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:06.201123    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-workload-socket\") pod \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") "
Sep 17 18:54:06.201229 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:06.201153    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-istiod-ca-cert\") pod \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") "
Sep 17 18:54:06.201229 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:06.201177    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-istio-envoy\") pod \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") "
Sep 17 18:54:06.201229 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:06.201218    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-istio-data\") pod \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") "
Sep 17 18:54:06.201492 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:06.201250    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8kqc\" (UniqueName: \"kubernetes.io/projected/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-kube-api-access-t8kqc\") pod \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") "
Sep 17 18:54:06.201492 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:06.201309    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-workload-certs\") pod \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") "
Sep 17 18:54:06.201492 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:06.201357    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-credential-socket\") pod \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\" (UID: \"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5\") "
Sep 17 18:54:06.202275 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:06.202222    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ebe139c6-bbeb-47bd-8966-4e3629fc6ce5" (UID: "ebe139c6-bbeb-47bd-8966-4e3629fc6ce5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:06.202493 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:06.202468    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ebe139c6-bbeb-47bd-8966-4e3629fc6ce5" (UID: "ebe139c6-bbeb-47bd-8966-4e3629fc6ce5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:06.202584 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:06.202557    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-istio-data" (OuterVolumeSpecName: "istio-data") pod "ebe139c6-bbeb-47bd-8966-4e3629fc6ce5" (UID: "ebe139c6-bbeb-47bd-8966-4e3629fc6ce5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:06.203259 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:06.203220    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ebe139c6-bbeb-47bd-8966-4e3629fc6ce5" (UID: "ebe139c6-bbeb-47bd-8966-4e3629fc6ce5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:54:06.203572 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:06.203549    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ebe139c6-bbeb-47bd-8966-4e3629fc6ce5" (UID: "ebe139c6-bbeb-47bd-8966-4e3629fc6ce5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:06.206433 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:06.206271    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ebe139c6-bbeb-47bd-8966-4e3629fc6ce5" (UID: "ebe139c6-bbeb-47bd-8966-4e3629fc6ce5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:54:06.206433 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:06.206324    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-kube-api-access-t8kqc" (OuterVolumeSpecName: "kube-api-access-t8kqc") pod "ebe139c6-bbeb-47bd-8966-4e3629fc6ce5" (UID: "ebe139c6-bbeb-47bd-8966-4e3629fc6ce5"). InnerVolumeSpecName "kube-api-access-t8kqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:54:06.206433 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:06.206350    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ebe139c6-bbeb-47bd-8966-4e3629fc6ce5" (UID: "ebe139c6-bbeb-47bd-8966-4e3629fc6ce5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 18:54:06.206433 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:06.206405    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-istio-token" (OuterVolumeSpecName: "istio-token") pod "ebe139c6-bbeb-47bd-8966-4e3629fc6ce5" (UID: "ebe139c6-bbeb-47bd-8966-4e3629fc6ce5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:54:06.302168 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:06.302119    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-workload-socket\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:54:06.302168 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:06.302165    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-istiod-ca-cert\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:54:06.302455 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:06.302181    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-istio-envoy\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:54:06.302455 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:06.302192    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-istio-data\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:54:06.302455 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:06.302201    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8kqc\" (UniqueName: \"kubernetes.io/projected/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-kube-api-access-t8kqc\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:54:06.302455 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:06.302212    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-workload-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:54:06.302455 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:06.302220    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-credential-socket\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:54:06.302455 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:06.302228    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-istio-token\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:54:06.302455 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:06.302236    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5-istio-podinfo\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:54:07.043988 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:07.043953    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh" event={"ID":"ebe139c6-bbeb-47bd-8966-4e3629fc6ce5","Type":"ContainerDied","Data":"5baa8a6ea1c105e7cd80f8b98f87c697d6c0995798aa1792c5258638e28e4562"}
Sep 17 18:54:07.044463 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:07.044014    2581 scope.go:117] "RemoveContainer" containerID="916265bb35139ba4a5f4ddb7703c8175c31a5e0ccbe600ffb9b826bfc788bea5"
Sep 17 18:54:07.044463 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:07.043967    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"
Sep 17 18:54:07.063367 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:07.063324    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"]
Sep 17 18:54:07.066755 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:07.066718    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-77d9dd58dd-cx7hh"]
Sep 17 18:54:08.358244 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:08.358206    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebe139c6-bbeb-47bd-8966-4e3629fc6ce5" path="/var/lib/kubelet/pods/ebe139c6-bbeb-47bd-8966-4e3629fc6ce5/volumes"
Sep 17 18:54:10.905438 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:10.905394    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"
Sep 17 18:54:10.906913 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:10.906869    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r" podUID="57699c84-5d0c-476b-93f8-e2dcb48524da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 17 18:54:20.905724 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:20.905667    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r" podUID="57699c84-5d0c-476b-93f8-e2dcb48524da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 17 18:54:30.906063 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:30.906016    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r" podUID="57699c84-5d0c-476b-93f8-e2dcb48524da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 17 18:54:40.906280 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:40.906219    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r" podUID="57699c84-5d0c-476b-93f8-e2dcb48524da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 17 18:54:50.906224 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:54:50.906174    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r" podUID="57699c84-5d0c-476b-93f8-e2dcb48524da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 17 18:55:00.905957 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:55:00.905881    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r" podUID="57699c84-5d0c-476b-93f8-e2dcb48524da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 17 18:55:10.905518 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:55:10.905466    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r" podUID="57699c84-5d0c-476b-93f8-e2dcb48524da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 17 18:55:13.274060 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:55:13.274020    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs"]
Sep 17 18:55:13.274635 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:55:13.274612    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebe139c6-bbeb-47bd-8966-4e3629fc6ce5" containerName="istio-proxy"
Sep 17 18:55:13.274711 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:55:13.274640    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebe139c6-bbeb-47bd-8966-4e3629fc6ce5" containerName="istio-proxy"
Sep 17 18:55:13.274711 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:55:13.274658    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebe139c6-bbeb-47bd-8966-4e3629fc6ce5" containerName="istio-proxy"
Sep 17 18:55:13.274711 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:55:13.274666    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebe139c6-bbeb-47bd-8966-4e3629fc6ce5" containerName="istio-proxy"
Sep 17 18:55:13.274812 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:55:13.274763    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebe139c6-bbeb-47bd-8966-4e3629fc6ce5" containerName="istio-proxy"
Sep 17 18:55:13.274812 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:55:13.274778    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebe139c6-bbeb-47bd-8966-4e3629fc6ce5" containerName="istio-proxy"
Sep 17 18:55:13.277176 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:55:13.277155    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs"
Sep 17 18:55:13.279895 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:55:13.279866    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 17 18:55:13.281006 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:55:13.280979    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 17 18:55:13.281144 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:55:13.281008    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 17 18:55:13.281144 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:55:13.281066    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-x4g28\""
Sep 17 18:55:13.281314 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:55:13.281293    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-jlw9b\""
Sep 17 18:55:13.284662 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:55:13.284630    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs"]
Sep 17 18:55:13.365066 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:55:13.365030    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/359e6794-5349-4804-b888-b71bc558f6f2-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs\" (UID: \"359e6794-5349-4804-b888-b71bc558f6f2\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs"
Sep 17 18:55:13.365285 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:55:13.365084    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5mzs\" (UniqueName: \"kubernetes.io/projected/359e6794-5349-4804-b888-b71bc558f6f2-kube-api-access-h5mzs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs\" (UID: \"359e6794-5349-4804-b888-b71bc558f6f2\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs"
Sep 17 18:55:13.465889 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:55:13.465838    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/359e6794-5349-4804-b888-b71bc558f6f2-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs\" (UID: \"359e6794-5349-4804-b888-b71bc558f6f2\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs"
Sep 17 18:55:13.466090 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:55:13.465948    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5mzs\" (UniqueName: \"kubernetes.io/projected/359e6794-5349-4804-b888-b71bc558f6f2-kube-api-access-h5mzs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs\" (UID: \"359e6794-5349-4804-b888-b71bc558f6f2\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs"
Sep 17 18:55:13.468559 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:55:13.468519    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/359e6794-5349-4804-b888-b71bc558f6f2-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs\" (UID: \"359e6794-5349-4804-b888-b71bc558f6f2\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs"
Sep 17 18:55:13.475037 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:55:13.475000    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5mzs\" (UniqueName: \"kubernetes.io/projected/359e6794-5349-4804-b888-b71bc558f6f2-kube-api-access-h5mzs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs\" (UID: \"359e6794-5349-4804-b888-b71bc558f6f2\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs"
Sep 17 18:55:13.589522 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:55:13.589412    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs"
Sep 17 18:55:13.865060 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:55:13.865029    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs"]
Sep 17 18:55:13.867058 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:55:13.867026    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod359e6794_5349_4804_b888_b71bc558f6f2.slice/crio-a8d3b2b3d2c712757fd82eedff0f56adbe15980fe46f89f3066f3c3bb435834f WatchSource:0}: Error finding container a8d3b2b3d2c712757fd82eedff0f56adbe15980fe46f89f3066f3c3bb435834f: Status 404 returned error can't find the container with id a8d3b2b3d2c712757fd82eedff0f56adbe15980fe46f89f3066f3c3bb435834f
Sep 17 18:55:14.395299 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:55:14.395259    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs" event={"ID":"359e6794-5349-4804-b888-b71bc558f6f2","Type":"ContainerStarted","Data":"8d15ad6a77194a4c0200fa39c03e4aaee01ed66795630cd33dfbf782d2805374"}
Sep 17 18:55:14.395792 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:55:14.395323    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs" event={"ID":"359e6794-5349-4804-b888-b71bc558f6f2","Type":"ContainerStarted","Data":"a8d3b2b3d2c712757fd82eedff0f56adbe15980fe46f89f3066f3c3bb435834f"}
Sep 17 18:55:14.395792 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:55:14.395349    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs"
Sep 17 18:55:14.414944 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:55:14.414850    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs" podStartSLOduration=1.414826559 podStartE2EDuration="1.414826559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:55:14.41357279 +0000 UTC m=+3502.504194442" watchObservedRunningTime="2026-09-17 18:55:14.414826559 +0000 UTC m=+3502.505448213"
Sep 17 18:55:20.905784 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:55:20.905693    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r" podUID="57699c84-5d0c-476b-93f8-e2dcb48524da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 17 18:55:30.905863 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:55:30.905816    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r" podUID="57699c84-5d0c-476b-93f8-e2dcb48524da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 17 18:55:40.906366 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:55:40.906311    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r" podUID="57699c84-5d0c-476b-93f8-e2dcb48524da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 17 18:55:45.402654 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:55:45.402619    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs"
Sep 17 18:55:50.906158 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:55:50.906106    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r" podUID="57699c84-5d0c-476b-93f8-e2dcb48524da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 17 18:56:00.916019 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:00.915985    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"
Sep 17 18:56:00.923960 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:00.923929    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"
Sep 17 18:56:05.322808 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:05.322772    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"]
Sep 17 18:56:20.345363 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:20.345279    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r" podUID="57699c84-5d0c-476b-93f8-e2dcb48524da" containerName="main" containerID="cri-o://f14759ad1c6de2c3fedbd19de1a1a59b6700f535c6da6018cad950e025ae0f20" gracePeriod=45
Sep 17 18:56:20.917072 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:20.917021    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r" podUID="57699c84-5d0c-476b-93f8-e2dcb48524da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Sep 17 18:56:21.100522 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.100494    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"
Sep 17 18:56:21.113187 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.113106    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67rtx\" (UniqueName: \"kubernetes.io/projected/57699c84-5d0c-476b-93f8-e2dcb48524da-kube-api-access-67rtx\") pod \"57699c84-5d0c-476b-93f8-e2dcb48524da\" (UID: \"57699c84-5d0c-476b-93f8-e2dcb48524da\") "
Sep 17 18:56:21.113187 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.113155    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57699c84-5d0c-476b-93f8-e2dcb48524da-kserve-provision-location\") pod \"57699c84-5d0c-476b-93f8-e2dcb48524da\" (UID: \"57699c84-5d0c-476b-93f8-e2dcb48524da\") "
Sep 17 18:56:21.113369 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.113187    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57699c84-5d0c-476b-93f8-e2dcb48524da-tmp-dir\") pod \"57699c84-5d0c-476b-93f8-e2dcb48524da\" (UID: \"57699c84-5d0c-476b-93f8-e2dcb48524da\") "
Sep 17 18:56:21.113369 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.113295    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57699c84-5d0c-476b-93f8-e2dcb48524da-model-cache\") pod \"57699c84-5d0c-476b-93f8-e2dcb48524da\" (UID: \"57699c84-5d0c-476b-93f8-e2dcb48524da\") "
Sep 17 18:56:21.113605 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.113574    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57699c84-5d0c-476b-93f8-e2dcb48524da-model-cache" (OuterVolumeSpecName: "model-cache") pod "57699c84-5d0c-476b-93f8-e2dcb48524da" (UID: "57699c84-5d0c-476b-93f8-e2dcb48524da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:21.115568 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.115542    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57699c84-5d0c-476b-93f8-e2dcb48524da-kube-api-access-67rtx" (OuterVolumeSpecName: "kube-api-access-67rtx") pod "57699c84-5d0c-476b-93f8-e2dcb48524da" (UID: "57699c84-5d0c-476b-93f8-e2dcb48524da"). InnerVolumeSpecName "kube-api-access-67rtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:56:21.126177 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.126148    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57699c84-5d0c-476b-93f8-e2dcb48524da-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "57699c84-5d0c-476b-93f8-e2dcb48524da" (UID: "57699c84-5d0c-476b-93f8-e2dcb48524da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:21.138727 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.138686    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57699c84-5d0c-476b-93f8-e2dcb48524da-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "57699c84-5d0c-476b-93f8-e2dcb48524da" (UID: "57699c84-5d0c-476b-93f8-e2dcb48524da"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:21.214364 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.214318    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57699c84-5d0c-476b-93f8-e2dcb48524da-tls-certs\") pod \"57699c84-5d0c-476b-93f8-e2dcb48524da\" (UID: \"57699c84-5d0c-476b-93f8-e2dcb48524da\") "
Sep 17 18:56:21.214364 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.214372    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57699c84-5d0c-476b-93f8-e2dcb48524da-dshm\") pod \"57699c84-5d0c-476b-93f8-e2dcb48524da\" (UID: \"57699c84-5d0c-476b-93f8-e2dcb48524da\") "
Sep 17 18:56:21.214637 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.214395    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57699c84-5d0c-476b-93f8-e2dcb48524da-home\") pod \"57699c84-5d0c-476b-93f8-e2dcb48524da\" (UID: \"57699c84-5d0c-476b-93f8-e2dcb48524da\") "
Sep 17 18:56:21.214637 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.214476    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57699c84-5d0c-476b-93f8-e2dcb48524da-cabundle-cert\") pod \"57699c84-5d0c-476b-93f8-e2dcb48524da\" (UID: \"57699c84-5d0c-476b-93f8-e2dcb48524da\") "
Sep 17 18:56:21.214754 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.214686    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67rtx\" (UniqueName: \"kubernetes.io/projected/57699c84-5d0c-476b-93f8-e2dcb48524da-kube-api-access-67rtx\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:56:21.214754 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.214705    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57699c84-5d0c-476b-93f8-e2dcb48524da-kserve-provision-location\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:56:21.214754 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.214720    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57699c84-5d0c-476b-93f8-e2dcb48524da-tmp-dir\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:56:21.214754 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.214735    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57699c84-5d0c-476b-93f8-e2dcb48524da-model-cache\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:56:21.215010 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.214881    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57699c84-5d0c-476b-93f8-e2dcb48524da-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "57699c84-5d0c-476b-93f8-e2dcb48524da" (UID: "57699c84-5d0c-476b-93f8-e2dcb48524da"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:56:21.215091 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.215062    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57699c84-5d0c-476b-93f8-e2dcb48524da-home" (OuterVolumeSpecName: "home") pod "57699c84-5d0c-476b-93f8-e2dcb48524da" (UID: "57699c84-5d0c-476b-93f8-e2dcb48524da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:21.216559 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.216536    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57699c84-5d0c-476b-93f8-e2dcb48524da-dshm" (OuterVolumeSpecName: "dshm") pod "57699c84-5d0c-476b-93f8-e2dcb48524da" (UID: "57699c84-5d0c-476b-93f8-e2dcb48524da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:56:21.216661 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.216587    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57699c84-5d0c-476b-93f8-e2dcb48524da-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "57699c84-5d0c-476b-93f8-e2dcb48524da" (UID: "57699c84-5d0c-476b-93f8-e2dcb48524da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:56:21.315359 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.315324    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57699c84-5d0c-476b-93f8-e2dcb48524da-cabundle-cert\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:56:21.315359 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.315355    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57699c84-5d0c-476b-93f8-e2dcb48524da-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:56:21.315359 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.315364    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57699c84-5d0c-476b-93f8-e2dcb48524da-dshm\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:56:21.315359 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.315371    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57699c84-5d0c-476b-93f8-e2dcb48524da-home\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:56:21.723244 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.723208    2581 generic.go:356] "Generic (PLEG): container finished" podID="57699c84-5d0c-476b-93f8-e2dcb48524da" containerID="f14759ad1c6de2c3fedbd19de1a1a59b6700f535c6da6018cad950e025ae0f20" exitCode=0
Sep 17 18:56:21.723650 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.723263    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"
Sep 17 18:56:21.723650 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.723297    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r" event={"ID":"57699c84-5d0c-476b-93f8-e2dcb48524da","Type":"ContainerDied","Data":"f14759ad1c6de2c3fedbd19de1a1a59b6700f535c6da6018cad950e025ae0f20"}
Sep 17 18:56:21.723650 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.723355    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r" event={"ID":"57699c84-5d0c-476b-93f8-e2dcb48524da","Type":"ContainerDied","Data":"3c7c5c3878a78e5e5effb52b6d8fdbf1fbf78b34a2dfc94cc57e1a757aeec1a9"}
Sep 17 18:56:21.723650 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.723379    2581 scope.go:117] "RemoveContainer" containerID="f14759ad1c6de2c3fedbd19de1a1a59b6700f535c6da6018cad950e025ae0f20"
Sep 17 18:56:21.732486 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.732465    2581 scope.go:117] "RemoveContainer" containerID="0d21d4fb0cd8903902b429a22d8fc1318099fc41e7193a20d9e9672424e12c9e"
Sep 17 18:56:21.740774 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.740755    2581 scope.go:117] "RemoveContainer" containerID="f14759ad1c6de2c3fedbd19de1a1a59b6700f535c6da6018cad950e025ae0f20"
Sep 17 18:56:21.741093 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:56:21.741073    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f14759ad1c6de2c3fedbd19de1a1a59b6700f535c6da6018cad950e025ae0f20\": container with ID starting with f14759ad1c6de2c3fedbd19de1a1a59b6700f535c6da6018cad950e025ae0f20 not found: ID does not exist" containerID="f14759ad1c6de2c3fedbd19de1a1a59b6700f535c6da6018cad950e025ae0f20"
Sep 17 18:56:21.741154 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.741104    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f14759ad1c6de2c3fedbd19de1a1a59b6700f535c6da6018cad950e025ae0f20"} err="failed to get container status \"f14759ad1c6de2c3fedbd19de1a1a59b6700f535c6da6018cad950e025ae0f20\": rpc error: code = NotFound desc = could not find container \"f14759ad1c6de2c3fedbd19de1a1a59b6700f535c6da6018cad950e025ae0f20\": container with ID starting with f14759ad1c6de2c3fedbd19de1a1a59b6700f535c6da6018cad950e025ae0f20 not found: ID does not exist"
Sep 17 18:56:21.741154 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.741122    2581 scope.go:117] "RemoveContainer" containerID="0d21d4fb0cd8903902b429a22d8fc1318099fc41e7193a20d9e9672424e12c9e"
Sep 17 18:56:21.741407 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:56:21.741389    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d21d4fb0cd8903902b429a22d8fc1318099fc41e7193a20d9e9672424e12c9e\": container with ID starting with 0d21d4fb0cd8903902b429a22d8fc1318099fc41e7193a20d9e9672424e12c9e not found: ID does not exist" containerID="0d21d4fb0cd8903902b429a22d8fc1318099fc41e7193a20d9e9672424e12c9e"
Sep 17 18:56:21.741449 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.741413    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d21d4fb0cd8903902b429a22d8fc1318099fc41e7193a20d9e9672424e12c9e"} err="failed to get container status \"0d21d4fb0cd8903902b429a22d8fc1318099fc41e7193a20d9e9672424e12c9e\": rpc error: code = NotFound desc = could not find container \"0d21d4fb0cd8903902b429a22d8fc1318099fc41e7193a20d9e9672424e12c9e\": container with ID starting with 0d21d4fb0cd8903902b429a22d8fc1318099fc41e7193a20d9e9672424e12c9e not found: ID does not exist"
Sep 17 18:56:21.747617 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.747594    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"]
Sep 17 18:56:21.750708 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:21.750680    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7565b79b5bc6q6r"]
Sep 17 18:56:22.358027 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:22.357993    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57699c84-5d0c-476b-93f8-e2dcb48524da" path="/var/lib/kubelet/pods/57699c84-5d0c-476b-93f8-e2dcb48524da/volumes"
Sep 17 18:56:35.419080 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.419042    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 18:56:35.419556 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.419410    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57699c84-5d0c-476b-93f8-e2dcb48524da" containerName="main"
Sep 17 18:56:35.419556 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.419423    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="57699c84-5d0c-476b-93f8-e2dcb48524da" containerName="main"
Sep 17 18:56:35.419556 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.419441    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57699c84-5d0c-476b-93f8-e2dcb48524da" containerName="storage-initializer"
Sep 17 18:56:35.419556 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.419447    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="57699c84-5d0c-476b-93f8-e2dcb48524da" containerName="storage-initializer"
Sep 17 18:56:35.419556 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.419507    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="57699c84-5d0c-476b-93f8-e2dcb48524da" containerName="main"
Sep 17 18:56:35.424251 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.424231    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:56:35.427013 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.426989    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:56:35.427367 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.427345    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 17 18:56:35.427458 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.427363    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 17 18:56:35.427458 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.427363    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 17 18:56:35.428150 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.428130    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 17 18:56:35.428259 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.428242    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-fhnpt\""
Sep 17 18:56:35.428333 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.428316    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-ld4jx\""
Sep 17 18:56:35.433160 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.433128    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 18:56:35.504251 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.504207    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98"]
Sep 17 18:56:35.507702 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.507684    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98"
Sep 17 18:56:35.510354 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.510334    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-8vdk8\""
Sep 17 18:56:35.518684 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.518663    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98"]
Sep 17 18:56:35.539096 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.539072    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f381c4a-213e-45fe-848f-8263fdf27b4d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9f381c4a-213e-45fe-848f-8263fdf27b4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:56:35.539219 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.539157    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f381c4a-213e-45fe-848f-8263fdf27b4d-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9f381c4a-213e-45fe-848f-8263fdf27b4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:56:35.539219 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.539201    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwc9s\" (UniqueName: \"kubernetes.io/projected/c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0-kube-api-access-pwc9s\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98\" (UID: \"c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98"
Sep 17 18:56:35.539339 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.539228    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f381c4a-213e-45fe-848f-8263fdf27b4d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9f381c4a-213e-45fe-848f-8263fdf27b4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:56:35.539339 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.539257    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f381c4a-213e-45fe-848f-8263fdf27b4d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9f381c4a-213e-45fe-848f-8263fdf27b4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:56:35.539460 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.539407    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f381c4a-213e-45fe-848f-8263fdf27b4d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9f381c4a-213e-45fe-848f-8263fdf27b4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:56:35.539517 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.539464    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r9bq\" (UniqueName: \"kubernetes.io/projected/9f381c4a-213e-45fe-848f-8263fdf27b4d-kube-api-access-4r9bq\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9f381c4a-213e-45fe-848f-8263fdf27b4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:56:35.539572 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.539513    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f381c4a-213e-45fe-848f-8263fdf27b4d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9f381c4a-213e-45fe-848f-8263fdf27b4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:56:35.539572 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.539556    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f381c4a-213e-45fe-848f-8263fdf27b4d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9f381c4a-213e-45fe-848f-8263fdf27b4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:56:35.539736 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.539582    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98\" (UID: \"c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98"
Sep 17 18:56:35.640418 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.640383    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r9bq\" (UniqueName: \"kubernetes.io/projected/9f381c4a-213e-45fe-848f-8263fdf27b4d-kube-api-access-4r9bq\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9f381c4a-213e-45fe-848f-8263fdf27b4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:56:35.640604 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.640453    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f381c4a-213e-45fe-848f-8263fdf27b4d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9f381c4a-213e-45fe-848f-8263fdf27b4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:56:35.640672 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.640588    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f381c4a-213e-45fe-848f-8263fdf27b4d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9f381c4a-213e-45fe-848f-8263fdf27b4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:56:35.640672 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.640640    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98\" (UID: \"c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98"
Sep 17 18:56:35.640778 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.640696    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f381c4a-213e-45fe-848f-8263fdf27b4d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9f381c4a-213e-45fe-848f-8263fdf27b4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:56:35.640778 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.640718    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f381c4a-213e-45fe-848f-8263fdf27b4d-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9f381c4a-213e-45fe-848f-8263fdf27b4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:56:35.640778 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.640754    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwc9s\" (UniqueName: \"kubernetes.io/projected/c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0-kube-api-access-pwc9s\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98\" (UID: \"c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98"
Sep 17 18:56:35.640960 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.640832    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f381c4a-213e-45fe-848f-8263fdf27b4d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9f381c4a-213e-45fe-848f-8263fdf27b4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:56:35.640960 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.640832    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f381c4a-213e-45fe-848f-8263fdf27b4d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9f381c4a-213e-45fe-848f-8263fdf27b4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:56:35.640960 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.640883    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f381c4a-213e-45fe-848f-8263fdf27b4d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9f381c4a-213e-45fe-848f-8263fdf27b4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:56:35.641135 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.640986    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f381c4a-213e-45fe-848f-8263fdf27b4d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9f381c4a-213e-45fe-848f-8263fdf27b4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:56:35.641135 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.641070    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f381c4a-213e-45fe-848f-8263fdf27b4d-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9f381c4a-213e-45fe-848f-8263fdf27b4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:56:35.641539 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.641260    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f381c4a-213e-45fe-848f-8263fdf27b4d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9f381c4a-213e-45fe-848f-8263fdf27b4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:56:35.641539 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.641349    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f381c4a-213e-45fe-848f-8263fdf27b4d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9f381c4a-213e-45fe-848f-8263fdf27b4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:56:35.641846 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.641812    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f381c4a-213e-45fe-848f-8263fdf27b4d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9f381c4a-213e-45fe-848f-8263fdf27b4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:56:35.643171 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.643151    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f381c4a-213e-45fe-848f-8263fdf27b4d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9f381c4a-213e-45fe-848f-8263fdf27b4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:56:35.643593 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.643568    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98\" (UID: \"c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98"
Sep 17 18:56:35.643861 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.643845    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f381c4a-213e-45fe-848f-8263fdf27b4d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9f381c4a-213e-45fe-848f-8263fdf27b4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:56:35.648963 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.648931    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r9bq\" (UniqueName: \"kubernetes.io/projected/9f381c4a-213e-45fe-848f-8263fdf27b4d-kube-api-access-4r9bq\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"9f381c4a-213e-45fe-848f-8263fdf27b4d\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:56:35.649136 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.649117    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwc9s\" (UniqueName: \"kubernetes.io/projected/c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0-kube-api-access-pwc9s\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98\" (UID: \"c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98"
Sep 17 18:56:35.735851 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.735832    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:56:35.818374 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:35.818342    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98"
Sep 17 18:56:36.009365 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:36.009337    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 18:56:36.012358 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:56:36.012320    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f381c4a_213e_45fe_848f_8263fdf27b4d.slice/crio-72603ef760b706e3ce4ad4f8a3d249f8abf73c5f329d70cfde07d9de463c5f39 WatchSource:0}: Error finding container 72603ef760b706e3ce4ad4f8a3d249f8abf73c5f329d70cfde07d9de463c5f39: Status 404 returned error can't find the container with id 72603ef760b706e3ce4ad4f8a3d249f8abf73c5f329d70cfde07d9de463c5f39
Sep 17 18:56:36.094769 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:36.094738    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98"]
Sep 17 18:56:36.097685 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:56:36.097654    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc33398f5_8fac_4f89_b0ea_cdd60d4d5dc0.slice/crio-0835cebe91619f14b5e4a757adc87fbc265bcb37da59b947b9f539a4494ccd12 WatchSource:0}: Error finding container 0835cebe91619f14b5e4a757adc87fbc265bcb37da59b947b9f539a4494ccd12: Status 404 returned error can't find the container with id 0835cebe91619f14b5e4a757adc87fbc265bcb37da59b947b9f539a4494ccd12
Sep 17 18:56:36.797210 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:36.797162    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"9f381c4a-213e-45fe-848f-8263fdf27b4d","Type":"ContainerStarted","Data":"273f94de150176d6ef7635fd28cdd99352f93bba0ac24c17ad0ce1bf9bd32cde"}
Sep 17 18:56:36.797692 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:36.797243    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"9f381c4a-213e-45fe-848f-8263fdf27b4d","Type":"ContainerStarted","Data":"72603ef760b706e3ce4ad4f8a3d249f8abf73c5f329d70cfde07d9de463c5f39"}
Sep 17 18:56:36.798782 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:36.798752    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98" event={"ID":"c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0","Type":"ContainerStarted","Data":"7e01813543eea1feabc063bb6e955203448aae711a0cfd0d0a8318dda5407e4b"}
Sep 17 18:56:36.798921 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:36.798812    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98" event={"ID":"c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0","Type":"ContainerStarted","Data":"0835cebe91619f14b5e4a757adc87fbc265bcb37da59b947b9f539a4494ccd12"}
Sep 17 18:56:36.798921 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:36.798844    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98"
Sep 17 18:56:36.834445 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:36.834387    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98" podStartSLOduration=1.8343711 podStartE2EDuration="1.8343711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:56:36.833552491 +0000 UTC m=+3584.924174143" watchObservedRunningTime="2026-09-17 18:56:36.8343711 +0000 UTC m=+3584.924992750"
Sep 17 18:56:38.810405 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:38.810372    2581 generic.go:356] "Generic (PLEG): container finished" podID="9f381c4a-213e-45fe-848f-8263fdf27b4d" containerID="273f94de150176d6ef7635fd28cdd99352f93bba0ac24c17ad0ce1bf9bd32cde" exitCode=0
Sep 17 18:56:38.810802 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:38.810445    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"9f381c4a-213e-45fe-848f-8263fdf27b4d","Type":"ContainerDied","Data":"273f94de150176d6ef7635fd28cdd99352f93bba0ac24c17ad0ce1bf9bd32cde"}
Sep 17 18:56:39.827087 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:39.827051    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"9f381c4a-213e-45fe-848f-8263fdf27b4d","Type":"ContainerStarted","Data":"55d89bd11772c40ed1646cc836776efdc24503d2bb62768d3693a3bf6d50351f"}
Sep 17 18:56:39.849279 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:39.849222    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.494415093 podStartE2EDuration="4.849207563s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.35479247s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:56:39.847994474 +0000 UTC m=+3587.938616138" watchObservedRunningTime="2026-09-17 18:56:39.849207563 +0000 UTC m=+3587.939829291"
Sep 17 18:56:40.736542 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:40.736505    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:56:45.736823 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:45.736712    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:56:45.738354 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:45.738324    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="9f381c4a-213e-45fe-848f-8263fdf27b4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 17 18:56:52.672870 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:52.672839    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/2.log"
Sep 17 18:56:52.688781 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:52.688750    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/2.log"
Sep 17 18:56:55.737170 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:56:55.737121    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="9f381c4a-213e-45fe-848f-8263fdf27b4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 17 18:57:05.737018 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:57:05.736966    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="9f381c4a-213e-45fe-848f-8263fdf27b4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 17 18:57:07.806080 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:57:07.806043    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98"
Sep 17 18:57:15.736304 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:57:15.736258    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="9f381c4a-213e-45fe-848f-8263fdf27b4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 17 18:57:18.105490 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:57:18.105448    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs"]
Sep 17 18:57:25.403365 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:57:25.403320    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs" podUID="359e6794-5349-4804-b888-b71bc558f6f2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:57:25.736614 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:57:25.736574    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="9f381c4a-213e-45fe-848f-8263fdf27b4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 17 18:57:33.125825 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:57:33.125777    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs" podUID="359e6794-5349-4804-b888-b71bc558f6f2" containerName="main" containerID="cri-o://8d15ad6a77194a4c0200fa39c03e4aaee01ed66795630cd33dfbf782d2805374" gracePeriod=45
Sep 17 18:57:35.402684 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:57:35.402643    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs" podUID="359e6794-5349-4804-b888-b71bc558f6f2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:57:35.737216 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:57:35.737174    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="9f381c4a-213e-45fe-848f-8263fdf27b4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 17 18:57:45.402439 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:57:45.402393    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs" podUID="359e6794-5349-4804-b888-b71bc558f6f2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:57:45.402824 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:57:45.402512    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs"
Sep 17 18:57:45.736842 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:57:45.736791    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="9f381c4a-213e-45fe-848f-8263fdf27b4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 17 18:57:55.402702 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:57:55.402662    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs" podUID="359e6794-5349-4804-b888-b71bc558f6f2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:57:55.736850 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:57:55.736789    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="9f381c4a-213e-45fe-848f-8263fdf27b4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 17 18:58:03.258242 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:03.258207    2581 generic.go:356] "Generic (PLEG): container finished" podID="359e6794-5349-4804-b888-b71bc558f6f2" containerID="8d15ad6a77194a4c0200fa39c03e4aaee01ed66795630cd33dfbf782d2805374" exitCode=0
Sep 17 18:58:03.258626 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:03.258275    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs" event={"ID":"359e6794-5349-4804-b888-b71bc558f6f2","Type":"ContainerDied","Data":"8d15ad6a77194a4c0200fa39c03e4aaee01ed66795630cd33dfbf782d2805374"}
Sep 17 18:58:03.368978 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:03.367803    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs"
Sep 17 18:58:03.475525 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:03.475487    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/359e6794-5349-4804-b888-b71bc558f6f2-tls-certs\") pod \"359e6794-5349-4804-b888-b71bc558f6f2\" (UID: \"359e6794-5349-4804-b888-b71bc558f6f2\") "
Sep 17 18:58:03.475733 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:03.475534    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5mzs\" (UniqueName: \"kubernetes.io/projected/359e6794-5349-4804-b888-b71bc558f6f2-kube-api-access-h5mzs\") pod \"359e6794-5349-4804-b888-b71bc558f6f2\" (UID: \"359e6794-5349-4804-b888-b71bc558f6f2\") "
Sep 17 18:58:03.477882 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:03.477837    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/359e6794-5349-4804-b888-b71bc558f6f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "359e6794-5349-4804-b888-b71bc558f6f2" (UID: "359e6794-5349-4804-b888-b71bc558f6f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:58:03.478022 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:03.477914    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/359e6794-5349-4804-b888-b71bc558f6f2-kube-api-access-h5mzs" (OuterVolumeSpecName: "kube-api-access-h5mzs") pod "359e6794-5349-4804-b888-b71bc558f6f2" (UID: "359e6794-5349-4804-b888-b71bc558f6f2"). InnerVolumeSpecName "kube-api-access-h5mzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:58:03.576737 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:03.576690    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/359e6794-5349-4804-b888-b71bc558f6f2-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:58:03.576737 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:03.576731    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5mzs\" (UniqueName: \"kubernetes.io/projected/359e6794-5349-4804-b888-b71bc558f6f2-kube-api-access-h5mzs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:58:04.264563 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:04.264530    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs"
Sep 17 18:58:04.265023 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:04.264575    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs" event={"ID":"359e6794-5349-4804-b888-b71bc558f6f2","Type":"ContainerDied","Data":"a8d3b2b3d2c712757fd82eedff0f56adbe15980fe46f89f3066f3c3bb435834f"}
Sep 17 18:58:04.265023 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:04.264666    2581 scope.go:117] "RemoveContainer" containerID="8d15ad6a77194a4c0200fa39c03e4aaee01ed66795630cd33dfbf782d2805374"
Sep 17 18:58:04.288210 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:04.288177    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs"]
Sep 17 18:58:04.292787 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:04.292755    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-blbbs"]
Sep 17 18:58:04.357962 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:04.357926    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="359e6794-5349-4804-b888-b71bc558f6f2" path="/var/lib/kubelet/pods/359e6794-5349-4804-b888-b71bc558f6f2/volumes"
Sep 17 18:58:05.736504 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:05.736453    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="9f381c4a-213e-45fe-848f-8263fdf27b4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Sep 17 18:58:15.745765 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:15.745674    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:58:16.333463 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:16.333429    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:58:22.396890 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:22.396855    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp"]
Sep 17 18:58:22.397289 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:22.397239    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="359e6794-5349-4804-b888-b71bc558f6f2" containerName="main"
Sep 17 18:58:22.397289 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:22.397251    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="359e6794-5349-4804-b888-b71bc558f6f2" containerName="main"
Sep 17 18:58:22.397393 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:22.397308    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="359e6794-5349-4804-b888-b71bc558f6f2" containerName="main"
Sep 17 18:58:22.400346 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:22.400324    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp"
Sep 17 18:58:22.403401 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:22.403380    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 17 18:58:22.403513 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:22.403422    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 17 18:58:22.404979 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:22.404957    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-w74rb\""
Sep 17 18:58:22.405086 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:22.404963    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-6clkh\""
Sep 17 18:58:22.405086 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:22.405006    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 17 18:58:22.412168 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:22.412142    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp"]
Sep 17 18:58:22.555261 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:22.555225    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ea8c903-de2a-4586-87f6-4faf70e702a6-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp\" (UID: \"1ea8c903-de2a-4586-87f6-4faf70e702a6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp"
Sep 17 18:58:22.555433 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:22.555294    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7zwv\" (UniqueName: \"kubernetes.io/projected/1ea8c903-de2a-4586-87f6-4faf70e702a6-kube-api-access-v7zwv\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp\" (UID: \"1ea8c903-de2a-4586-87f6-4faf70e702a6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp"
Sep 17 18:58:22.656378 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:22.656281    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ea8c903-de2a-4586-87f6-4faf70e702a6-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp\" (UID: \"1ea8c903-de2a-4586-87f6-4faf70e702a6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp"
Sep 17 18:58:22.656378 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:22.656357    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7zwv\" (UniqueName: \"kubernetes.io/projected/1ea8c903-de2a-4586-87f6-4faf70e702a6-kube-api-access-v7zwv\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp\" (UID: \"1ea8c903-de2a-4586-87f6-4faf70e702a6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp"
Sep 17 18:58:22.658920 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:22.658879    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ea8c903-de2a-4586-87f6-4faf70e702a6-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp\" (UID: \"1ea8c903-de2a-4586-87f6-4faf70e702a6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp"
Sep 17 18:58:22.665624 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:22.665596    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7zwv\" (UniqueName: \"kubernetes.io/projected/1ea8c903-de2a-4586-87f6-4faf70e702a6-kube-api-access-v7zwv\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp\" (UID: \"1ea8c903-de2a-4586-87f6-4faf70e702a6\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp"
Sep 17 18:58:22.711424 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:22.711386    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp"
Sep 17 18:58:22.984887 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:22.984855    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp"]
Sep 17 18:58:22.988008 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:58:22.987972    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea8c903_de2a_4586_87f6_4faf70e702a6.slice/crio-dc6903d7f789368e4e0de5bea5b6785f78dd1e0e0a855470149f134f0fe59a00 WatchSource:0}: Error finding container dc6903d7f789368e4e0de5bea5b6785f78dd1e0e0a855470149f134f0fe59a00: Status 404 returned error can't find the container with id dc6903d7f789368e4e0de5bea5b6785f78dd1e0e0a855470149f134f0fe59a00
Sep 17 18:58:23.364261 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:23.364158    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp" event={"ID":"1ea8c903-de2a-4586-87f6-4faf70e702a6","Type":"ContainerStarted","Data":"248160cf193f9854112e1298d04d57aa33cfc95b1fc373043568c1bd140492b5"}
Sep 17 18:58:23.364261 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:23.364215    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp" event={"ID":"1ea8c903-de2a-4586-87f6-4faf70e702a6","Type":"ContainerStarted","Data":"dc6903d7f789368e4e0de5bea5b6785f78dd1e0e0a855470149f134f0fe59a00"}
Sep 17 18:58:23.364518 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:23.364303    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp"
Sep 17 18:58:23.399962 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:23.399887    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp" podStartSLOduration=1.399866456 podStartE2EDuration="1.399866456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:58:23.397416552 +0000 UTC m=+3691.488038203" watchObservedRunningTime="2026-09-17 18:58:23.399866456 +0000 UTC m=+3691.490488108"
Sep 17 18:58:36.293172 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:36.293129    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 18:58:36.297016 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:36.296976    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98"]
Sep 17 18:58:37.805791 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:37.805746    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98" podUID="c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:58:47.806539 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:47.806493    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98" podUID="c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:58:51.315726 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:51.315658    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98" podUID="c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0" containerName="main" containerID="cri-o://7e01813543eea1feabc063bb6e955203448aae711a0cfd0d0a8318dda5407e4b" gracePeriod=45
Sep 17 18:58:51.316213 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:51.316107    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="9f381c4a-213e-45fe-848f-8263fdf27b4d" containerName="main" containerID="cri-o://55d89bd11772c40ed1646cc836776efdc24503d2bb62768d3693a3bf6d50351f" gracePeriod=45
Sep 17 18:58:52.061703 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.061673    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:58:52.105075 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.104980    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f381c4a-213e-45fe-848f-8263fdf27b4d-cabundle-cert\") pod \"9f381c4a-213e-45fe-848f-8263fdf27b4d\" (UID: \"9f381c4a-213e-45fe-848f-8263fdf27b4d\") "
Sep 17 18:58:52.105075 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.105025    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f381c4a-213e-45fe-848f-8263fdf27b4d-kserve-provision-location\") pod \"9f381c4a-213e-45fe-848f-8263fdf27b4d\" (UID: \"9f381c4a-213e-45fe-848f-8263fdf27b4d\") "
Sep 17 18:58:52.105075 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.105062    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f381c4a-213e-45fe-848f-8263fdf27b4d-model-cache\") pod \"9f381c4a-213e-45fe-848f-8263fdf27b4d\" (UID: \"9f381c4a-213e-45fe-848f-8263fdf27b4d\") "
Sep 17 18:58:52.105351 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.105095    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f381c4a-213e-45fe-848f-8263fdf27b4d-tmp-dir\") pod \"9f381c4a-213e-45fe-848f-8263fdf27b4d\" (UID: \"9f381c4a-213e-45fe-848f-8263fdf27b4d\") "
Sep 17 18:58:52.105351 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.105138    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f381c4a-213e-45fe-848f-8263fdf27b4d-dshm\") pod \"9f381c4a-213e-45fe-848f-8263fdf27b4d\" (UID: \"9f381c4a-213e-45fe-848f-8263fdf27b4d\") "
Sep 17 18:58:52.105351 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.105160    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f381c4a-213e-45fe-848f-8263fdf27b4d-home\") pod \"9f381c4a-213e-45fe-848f-8263fdf27b4d\" (UID: \"9f381c4a-213e-45fe-848f-8263fdf27b4d\") "
Sep 17 18:58:52.105351 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.105214    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4r9bq\" (UniqueName: \"kubernetes.io/projected/9f381c4a-213e-45fe-848f-8263fdf27b4d-kube-api-access-4r9bq\") pod \"9f381c4a-213e-45fe-848f-8263fdf27b4d\" (UID: \"9f381c4a-213e-45fe-848f-8263fdf27b4d\") "
Sep 17 18:58:52.105351 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.105265    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f381c4a-213e-45fe-848f-8263fdf27b4d-tls-certs\") pod \"9f381c4a-213e-45fe-848f-8263fdf27b4d\" (UID: \"9f381c4a-213e-45fe-848f-8263fdf27b4d\") "
Sep 17 18:58:52.105631 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.105327    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f381c4a-213e-45fe-848f-8263fdf27b4d-model-cache" (OuterVolumeSpecName: "model-cache") pod "9f381c4a-213e-45fe-848f-8263fdf27b4d" (UID: "9f381c4a-213e-45fe-848f-8263fdf27b4d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:58:52.105631 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.105411    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f381c4a-213e-45fe-848f-8263fdf27b4d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9f381c4a-213e-45fe-848f-8263fdf27b4d" (UID: "9f381c4a-213e-45fe-848f-8263fdf27b4d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:58:52.105799 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.105743    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f381c4a-213e-45fe-848f-8263fdf27b4d-cabundle-cert\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:58:52.105799 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.105764    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f381c4a-213e-45fe-848f-8263fdf27b4d-model-cache\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:58:52.105995 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.105970    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f381c4a-213e-45fe-848f-8263fdf27b4d-home" (OuterVolumeSpecName: "home") pod "9f381c4a-213e-45fe-848f-8263fdf27b4d" (UID: "9f381c4a-213e-45fe-848f-8263fdf27b4d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:58:52.107891 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.107854    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f381c4a-213e-45fe-848f-8263fdf27b4d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9f381c4a-213e-45fe-848f-8263fdf27b4d" (UID: "9f381c4a-213e-45fe-848f-8263fdf27b4d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:58:52.108009 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.107918    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f381c4a-213e-45fe-848f-8263fdf27b4d-kube-api-access-4r9bq" (OuterVolumeSpecName: "kube-api-access-4r9bq") pod "9f381c4a-213e-45fe-848f-8263fdf27b4d" (UID: "9f381c4a-213e-45fe-848f-8263fdf27b4d"). InnerVolumeSpecName "kube-api-access-4r9bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:58:52.108009 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.107990    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f381c4a-213e-45fe-848f-8263fdf27b4d-dshm" (OuterVolumeSpecName: "dshm") pod "9f381c4a-213e-45fe-848f-8263fdf27b4d" (UID: "9f381c4a-213e-45fe-848f-8263fdf27b4d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:58:52.124272 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.124192    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f381c4a-213e-45fe-848f-8263fdf27b4d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9f381c4a-213e-45fe-848f-8263fdf27b4d" (UID: "9f381c4a-213e-45fe-848f-8263fdf27b4d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:58:52.137849 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.137806    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f381c4a-213e-45fe-848f-8263fdf27b4d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9f381c4a-213e-45fe-848f-8263fdf27b4d" (UID: "9f381c4a-213e-45fe-848f-8263fdf27b4d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:58:52.206388 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.206341    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r9bq\" (UniqueName: \"kubernetes.io/projected/9f381c4a-213e-45fe-848f-8263fdf27b4d-kube-api-access-4r9bq\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:58:52.206388 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.206384    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f381c4a-213e-45fe-848f-8263fdf27b4d-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:58:52.206591 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.206400    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f381c4a-213e-45fe-848f-8263fdf27b4d-kserve-provision-location\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:58:52.206591 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.206435    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f381c4a-213e-45fe-848f-8263fdf27b4d-tmp-dir\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:58:52.206591 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.206447    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f381c4a-213e-45fe-848f-8263fdf27b4d-dshm\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:58:52.206591 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.206458    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f381c4a-213e-45fe-848f-8263fdf27b4d-home\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:58:52.512812 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.512771    2581 generic.go:356] "Generic (PLEG): container finished" podID="9f381c4a-213e-45fe-848f-8263fdf27b4d" containerID="55d89bd11772c40ed1646cc836776efdc24503d2bb62768d3693a3bf6d50351f" exitCode=0
Sep 17 18:58:52.513313 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.512827    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 17 18:58:52.513313 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.512826    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"9f381c4a-213e-45fe-848f-8263fdf27b4d","Type":"ContainerDied","Data":"55d89bd11772c40ed1646cc836776efdc24503d2bb62768d3693a3bf6d50351f"}
Sep 17 18:58:52.513313 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.512968    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"9f381c4a-213e-45fe-848f-8263fdf27b4d","Type":"ContainerDied","Data":"72603ef760b706e3ce4ad4f8a3d249f8abf73c5f329d70cfde07d9de463c5f39"}
Sep 17 18:58:52.513313 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.513005    2581 scope.go:117] "RemoveContainer" containerID="55d89bd11772c40ed1646cc836776efdc24503d2bb62768d3693a3bf6d50351f"
Sep 17 18:58:52.521726 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.521702    2581 scope.go:117] "RemoveContainer" containerID="273f94de150176d6ef7635fd28cdd99352f93bba0ac24c17ad0ce1bf9bd32cde"
Sep 17 18:58:52.530329 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.530309    2581 scope.go:117] "RemoveContainer" containerID="55d89bd11772c40ed1646cc836776efdc24503d2bb62768d3693a3bf6d50351f"
Sep 17 18:58:52.530435 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.530411    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 18:58:52.530631 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:58:52.530614    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55d89bd11772c40ed1646cc836776efdc24503d2bb62768d3693a3bf6d50351f\": container with ID starting with 55d89bd11772c40ed1646cc836776efdc24503d2bb62768d3693a3bf6d50351f not found: ID does not exist" containerID="55d89bd11772c40ed1646cc836776efdc24503d2bb62768d3693a3bf6d50351f"
Sep 17 18:58:52.530677 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.530643    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55d89bd11772c40ed1646cc836776efdc24503d2bb62768d3693a3bf6d50351f"} err="failed to get container status \"55d89bd11772c40ed1646cc836776efdc24503d2bb62768d3693a3bf6d50351f\": rpc error: code = NotFound desc = could not find container \"55d89bd11772c40ed1646cc836776efdc24503d2bb62768d3693a3bf6d50351f\": container with ID starting with 55d89bd11772c40ed1646cc836776efdc24503d2bb62768d3693a3bf6d50351f not found: ID does not exist"
Sep 17 18:58:52.530677 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.530664    2581 scope.go:117] "RemoveContainer" containerID="273f94de150176d6ef7635fd28cdd99352f93bba0ac24c17ad0ce1bf9bd32cde"
Sep 17 18:58:52.531005 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:58:52.530984    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"273f94de150176d6ef7635fd28cdd99352f93bba0ac24c17ad0ce1bf9bd32cde\": container with ID starting with 273f94de150176d6ef7635fd28cdd99352f93bba0ac24c17ad0ce1bf9bd32cde not found: ID does not exist" containerID="273f94de150176d6ef7635fd28cdd99352f93bba0ac24c17ad0ce1bf9bd32cde"
Sep 17 18:58:52.531077 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.531009    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273f94de150176d6ef7635fd28cdd99352f93bba0ac24c17ad0ce1bf9bd32cde"} err="failed to get container status \"273f94de150176d6ef7635fd28cdd99352f93bba0ac24c17ad0ce1bf9bd32cde\": rpc error: code = NotFound desc = could not find container \"273f94de150176d6ef7635fd28cdd99352f93bba0ac24c17ad0ce1bf9bd32cde\": container with ID starting with 273f94de150176d6ef7635fd28cdd99352f93bba0ac24c17ad0ce1bf9bd32cde not found: ID does not exist"
Sep 17 18:58:52.536176 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:52.536153    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 17 18:58:54.359127 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:54.359088    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f381c4a-213e-45fe-848f-8263fdf27b4d" path="/var/lib/kubelet/pods/9f381c4a-213e-45fe-848f-8263fdf27b4d/volumes"
Sep 17 18:58:54.370359 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:54.370328    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp"
Sep 17 18:58:57.806572 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:57.806531    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98" podUID="c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:58:57.807066 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:58:57.806646    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98"
Sep 17 18:59:07.806166 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:07.806127    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98" podUID="c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:59:17.805989 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:17.805946    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98" podUID="c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:59:21.562413 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:21.562384    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98"
Sep 17 18:59:21.574568 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:21.574502    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0-tls-certs\") pod \"c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0\" (UID: \"c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0\") "
Sep 17 18:59:21.574677 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:21.574601    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwc9s\" (UniqueName: \"kubernetes.io/projected/c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0-kube-api-access-pwc9s\") pod \"c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0\" (UID: \"c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0\") "
Sep 17 18:59:21.576535 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:21.576494    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0" (UID: "c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:59:21.576625 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:21.576597    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0-kube-api-access-pwc9s" (OuterVolumeSpecName: "kube-api-access-pwc9s") pod "c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0" (UID: "c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0"). InnerVolumeSpecName "kube-api-access-pwc9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:59:21.651502 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:21.651465    2581 generic.go:356] "Generic (PLEG): container finished" podID="c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0" containerID="7e01813543eea1feabc063bb6e955203448aae711a0cfd0d0a8318dda5407e4b" exitCode=0
Sep 17 18:59:21.651667 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:21.651513    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98"
Sep 17 18:59:21.651667 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:21.651553    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98" event={"ID":"c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0","Type":"ContainerDied","Data":"7e01813543eea1feabc063bb6e955203448aae711a0cfd0d0a8318dda5407e4b"}
Sep 17 18:59:21.651667 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:21.651610    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98" event={"ID":"c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0","Type":"ContainerDied","Data":"0835cebe91619f14b5e4a757adc87fbc265bcb37da59b947b9f539a4494ccd12"}
Sep 17 18:59:21.651667 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:21.651635    2581 scope.go:117] "RemoveContainer" containerID="7e01813543eea1feabc063bb6e955203448aae711a0cfd0d0a8318dda5407e4b"
Sep 17 18:59:21.663058 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:21.663029    2581 scope.go:117] "RemoveContainer" containerID="7e01813543eea1feabc063bb6e955203448aae711a0cfd0d0a8318dda5407e4b"
Sep 17 18:59:21.663593 ip-10-0-130-28 kubenswrapper[2581]: E0917 18:59:21.663501    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e01813543eea1feabc063bb6e955203448aae711a0cfd0d0a8318dda5407e4b\": container with ID starting with 7e01813543eea1feabc063bb6e955203448aae711a0cfd0d0a8318dda5407e4b not found: ID does not exist" containerID="7e01813543eea1feabc063bb6e955203448aae711a0cfd0d0a8318dda5407e4b"
Sep 17 18:59:21.663593 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:21.663542    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e01813543eea1feabc063bb6e955203448aae711a0cfd0d0a8318dda5407e4b"} err="failed to get container status \"7e01813543eea1feabc063bb6e955203448aae711a0cfd0d0a8318dda5407e4b\": rpc error: code = NotFound desc = could not find container \"7e01813543eea1feabc063bb6e955203448aae711a0cfd0d0a8318dda5407e4b\": container with ID starting with 7e01813543eea1feabc063bb6e955203448aae711a0cfd0d0a8318dda5407e4b not found: ID does not exist"
Sep 17 18:59:21.675933 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:21.675885    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwc9s\" (UniqueName: \"kubernetes.io/projected/c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0-kube-api-access-pwc9s\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:59:21.676092 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:21.676068    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 18:59:21.678076 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:21.678054    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98"]
Sep 17 18:59:21.680926 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:21.680892    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schex9f98"]
Sep 17 18:59:22.358553 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:22.358518    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0" path="/var/lib/kubelet/pods/c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0/volumes"
Sep 17 18:59:40.471355 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.471317    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w"]
Sep 17 18:59:40.472006 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.471762    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0" containerName="main"
Sep 17 18:59:40.472006 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.471778    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0" containerName="main"
Sep 17 18:59:40.472006 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.471805    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f381c4a-213e-45fe-848f-8263fdf27b4d" containerName="storage-initializer"
Sep 17 18:59:40.472006 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.471811    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f381c4a-213e-45fe-848f-8263fdf27b4d" containerName="storage-initializer"
Sep 17 18:59:40.472006 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.471821    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f381c4a-213e-45fe-848f-8263fdf27b4d" containerName="main"
Sep 17 18:59:40.472006 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.471829    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f381c4a-213e-45fe-848f-8263fdf27b4d" containerName="main"
Sep 17 18:59:40.472006 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.471893    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c33398f5-8fac-4f89-b0ea-cdd60d4d5dc0" containerName="main"
Sep 17 18:59:40.472006 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.471917    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f381c4a-213e-45fe-848f-8263fdf27b4d" containerName="main"
Sep 17 18:59:40.474549 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.474530    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w"
Sep 17 18:59:40.478348 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.478319    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-p6fxs\""
Sep 17 18:59:40.478495 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.478351    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 17 18:59:40.478495 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.478418    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 17 18:59:40.478495 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.478431    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 17 18:59:40.484791 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.484762    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w"]
Sep 17 18:59:40.547168 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.547129    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be677a1a-5dc5-408e-ae70-109731d63381-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-cm55w\" (UID: \"be677a1a-5dc5-408e-ae70-109731d63381\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w"
Sep 17 18:59:40.547168 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.547183    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be677a1a-5dc5-408e-ae70-109731d63381-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-cm55w\" (UID: \"be677a1a-5dc5-408e-ae70-109731d63381\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w"
Sep 17 18:59:40.547436 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.547248    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5txhc\" (UniqueName: \"kubernetes.io/projected/be677a1a-5dc5-408e-ae70-109731d63381-kube-api-access-5txhc\") pod \"scheduler-inline-config-test-kserve-8655bc757-cm55w\" (UID: \"be677a1a-5dc5-408e-ae70-109731d63381\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w"
Sep 17 18:59:40.547436 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.547328    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be677a1a-5dc5-408e-ae70-109731d63381-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-cm55w\" (UID: \"be677a1a-5dc5-408e-ae70-109731d63381\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w"
Sep 17 18:59:40.547436 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.547371    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be677a1a-5dc5-408e-ae70-109731d63381-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-cm55w\" (UID: \"be677a1a-5dc5-408e-ae70-109731d63381\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w"
Sep 17 18:59:40.547436 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.547399    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be677a1a-5dc5-408e-ae70-109731d63381-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-cm55w\" (UID: \"be677a1a-5dc5-408e-ae70-109731d63381\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w"
Sep 17 18:59:40.648880 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.648822    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be677a1a-5dc5-408e-ae70-109731d63381-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-cm55w\" (UID: \"be677a1a-5dc5-408e-ae70-109731d63381\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w"
Sep 17 18:59:40.648880 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.648889    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be677a1a-5dc5-408e-ae70-109731d63381-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-cm55w\" (UID: \"be677a1a-5dc5-408e-ae70-109731d63381\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w"
Sep 17 18:59:40.649155 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.648944    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be677a1a-5dc5-408e-ae70-109731d63381-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-cm55w\" (UID: \"be677a1a-5dc5-408e-ae70-109731d63381\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w"
Sep 17 18:59:40.649155 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.648989    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be677a1a-5dc5-408e-ae70-109731d63381-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-cm55w\" (UID: \"be677a1a-5dc5-408e-ae70-109731d63381\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w"
Sep 17 18:59:40.649155 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.649027    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be677a1a-5dc5-408e-ae70-109731d63381-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-cm55w\" (UID: \"be677a1a-5dc5-408e-ae70-109731d63381\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w"
Sep 17 18:59:40.649155 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.649066    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5txhc\" (UniqueName: \"kubernetes.io/projected/be677a1a-5dc5-408e-ae70-109731d63381-kube-api-access-5txhc\") pod \"scheduler-inline-config-test-kserve-8655bc757-cm55w\" (UID: \"be677a1a-5dc5-408e-ae70-109731d63381\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w"
Sep 17 18:59:40.649381 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.649327    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be677a1a-5dc5-408e-ae70-109731d63381-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-cm55w\" (UID: \"be677a1a-5dc5-408e-ae70-109731d63381\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w"
Sep 17 18:59:40.649381 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.649345    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be677a1a-5dc5-408e-ae70-109731d63381-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-cm55w\" (UID: \"be677a1a-5dc5-408e-ae70-109731d63381\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w"
Sep 17 18:59:40.649478 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.649400    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be677a1a-5dc5-408e-ae70-109731d63381-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-cm55w\" (UID: \"be677a1a-5dc5-408e-ae70-109731d63381\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w"
Sep 17 18:59:40.651274 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.651252    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be677a1a-5dc5-408e-ae70-109731d63381-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-cm55w\" (UID: \"be677a1a-5dc5-408e-ae70-109731d63381\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w"
Sep 17 18:59:40.651540 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.651524    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be677a1a-5dc5-408e-ae70-109731d63381-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-cm55w\" (UID: \"be677a1a-5dc5-408e-ae70-109731d63381\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w"
Sep 17 18:59:40.657350 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.657331    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5txhc\" (UniqueName: \"kubernetes.io/projected/be677a1a-5dc5-408e-ae70-109731d63381-kube-api-access-5txhc\") pod \"scheduler-inline-config-test-kserve-8655bc757-cm55w\" (UID: \"be677a1a-5dc5-408e-ae70-109731d63381\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w"
Sep 17 18:59:40.786096 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:40.786008    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w"
Sep 17 18:59:41.057154 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:41.057071    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w"]
Sep 17 18:59:41.059455 ip-10-0-130-28 kubenswrapper[2581]: W0917 18:59:41.059419    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe677a1a_5dc5_408e_ae70_109731d63381.slice/crio-97765e5d86e5a84b4e058f9a7c012796310f7ec12e26669938f3a606f819ef5f WatchSource:0}: Error finding container 97765e5d86e5a84b4e058f9a7c012796310f7ec12e26669938f3a606f819ef5f: Status 404 returned error can't find the container with id 97765e5d86e5a84b4e058f9a7c012796310f7ec12e26669938f3a606f819ef5f
Sep 17 18:59:41.756742 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:41.756696    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w" event={"ID":"be677a1a-5dc5-408e-ae70-109731d63381","Type":"ContainerStarted","Data":"24ce45c470826e6afb09141ac16753296b0c691079cb7996ac8645c5ea942438"}
Sep 17 18:59:41.757274 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:41.756800    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w" event={"ID":"be677a1a-5dc5-408e-ae70-109731d63381","Type":"ContainerStarted","Data":"97765e5d86e5a84b4e058f9a7c012796310f7ec12e26669938f3a606f819ef5f"}
Sep 17 18:59:41.775673 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:41.775612    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w" podStartSLOduration=1.775596746 podStartE2EDuration="1.775596746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:59:41.774645872 +0000 UTC m=+3769.865267539" watchObservedRunningTime="2026-09-17 18:59:41.775596746 +0000 UTC m=+3769.866218394"
Sep 17 18:59:41.786915 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:41.786870    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w"
Sep 17 18:59:50.786747 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:50.786640    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w"
Sep 17 18:59:50.799988 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:50.799959    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w"
Sep 17 18:59:50.810861 ip-10-0-130-28 kubenswrapper[2581]: I0917 18:59:50.810835    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w"
Sep 17 19:00:14.608010 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:14.607972    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w"]
Sep 17 19:00:14.617481 ip-10-0-130-28 kubenswrapper[2581]: E0917 19:00:14.617434    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:00:14.617481 ip-10-0-130-28 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:00:14.617481 ip-10-0-130-28 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 19:00:14.617481 ip-10-0-130-28 kubenswrapper[2581]:  > containerID="24ce45c470826e6afb09141ac16753296b0c691079cb7996ac8645c5ea942438" cmd=["/bin/sleep","15"]
Sep 17 19:00:14.617728 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:14.617504    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w" message="" err=<
Sep 17 19:00:14.617728 ip-10-0-130-28 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:00:14.617728 ip-10-0-130-28 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 19:00:14.617728 ip-10-0-130-28 kubenswrapper[2581]:  >
Sep 17 19:00:14.617728 ip-10-0-130-28 kubenswrapper[2581]: E0917 19:00:14.617520    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:00:14.617728 ip-10-0-130-28 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:00:14.617728 ip-10-0-130-28 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 19:00:14.617728 ip-10-0-130-28 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w" podUID="be677a1a-5dc5-408e-ae70-109731d63381" containerName="main" containerID="cri-o://24ce45c470826e6afb09141ac16753296b0c691079cb7996ac8645c5ea942438"
Sep 17 19:00:14.617728 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:14.617579    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w" podUID="be677a1a-5dc5-408e-ae70-109731d63381" containerName="main" containerID="cri-o://24ce45c470826e6afb09141ac16753296b0c691079cb7996ac8645c5ea942438" gracePeriod=60
Sep 17 19:00:14.800913 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:14.800863    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w" podUID="be677a1a-5dc5-408e-ae70-109731d63381" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": dial tcp 10.133.0.80:8000: connect: connection refused"
Sep 17 19:00:14.916213 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:14.916180    2581 generic.go:356] "Generic (PLEG): container finished" podID="be677a1a-5dc5-408e-ae70-109731d63381" containerID="24ce45c470826e6afb09141ac16753296b0c691079cb7996ac8645c5ea942438" exitCode=0
Sep 17 19:00:14.916383 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:14.916246    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w" event={"ID":"be677a1a-5dc5-408e-ae70-109731d63381","Type":"ContainerDied","Data":"24ce45c470826e6afb09141ac16753296b0c691079cb7996ac8645c5ea942438"}
Sep 17 19:00:14.987825 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:14.987797    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w"
Sep 17 19:00:15.065803 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:15.065760    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be677a1a-5dc5-408e-ae70-109731d63381-home\") pod \"be677a1a-5dc5-408e-ae70-109731d63381\" (UID: \"be677a1a-5dc5-408e-ae70-109731d63381\") "
Sep 17 19:00:15.066030 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:15.065826    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be677a1a-5dc5-408e-ae70-109731d63381-tmp-dir\") pod \"be677a1a-5dc5-408e-ae70-109731d63381\" (UID: \"be677a1a-5dc5-408e-ae70-109731d63381\") "
Sep 17 19:00:15.066030 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:15.065930    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5txhc\" (UniqueName: \"kubernetes.io/projected/be677a1a-5dc5-408e-ae70-109731d63381-kube-api-access-5txhc\") pod \"be677a1a-5dc5-408e-ae70-109731d63381\" (UID: \"be677a1a-5dc5-408e-ae70-109731d63381\") "
Sep 17 19:00:15.066030 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:15.066007    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be677a1a-5dc5-408e-ae70-109731d63381-tls-certs\") pod \"be677a1a-5dc5-408e-ae70-109731d63381\" (UID: \"be677a1a-5dc5-408e-ae70-109731d63381\") "
Sep 17 19:00:15.066205 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:15.066043    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be677a1a-5dc5-408e-ae70-109731d63381-dshm\") pod \"be677a1a-5dc5-408e-ae70-109731d63381\" (UID: \"be677a1a-5dc5-408e-ae70-109731d63381\") "
Sep 17 19:00:15.066205 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:15.066083    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be677a1a-5dc5-408e-ae70-109731d63381-model-cache\") pod \"be677a1a-5dc5-408e-ae70-109731d63381\" (UID: \"be677a1a-5dc5-408e-ae70-109731d63381\") "
Sep 17 19:00:15.066205 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:15.066109    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be677a1a-5dc5-408e-ae70-109731d63381-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "be677a1a-5dc5-408e-ae70-109731d63381" (UID: "be677a1a-5dc5-408e-ae70-109731d63381"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:00:15.066205 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:15.066124    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be677a1a-5dc5-408e-ae70-109731d63381-home" (OuterVolumeSpecName: "home") pod "be677a1a-5dc5-408e-ae70-109731d63381" (UID: "be677a1a-5dc5-408e-ae70-109731d63381"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:00:15.066411 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:15.066341    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be677a1a-5dc5-408e-ae70-109731d63381-model-cache" (OuterVolumeSpecName: "model-cache") pod "be677a1a-5dc5-408e-ae70-109731d63381" (UID: "be677a1a-5dc5-408e-ae70-109731d63381"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:00:15.066411 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:15.066358    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be677a1a-5dc5-408e-ae70-109731d63381-home\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 19:00:15.066411 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:15.066399    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be677a1a-5dc5-408e-ae70-109731d63381-tmp-dir\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 19:00:15.068290 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:15.068259    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be677a1a-5dc5-408e-ae70-109731d63381-dshm" (OuterVolumeSpecName: "dshm") pod "be677a1a-5dc5-408e-ae70-109731d63381" (UID: "be677a1a-5dc5-408e-ae70-109731d63381"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:00:15.068397 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:15.068288    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be677a1a-5dc5-408e-ae70-109731d63381-kube-api-access-5txhc" (OuterVolumeSpecName: "kube-api-access-5txhc") pod "be677a1a-5dc5-408e-ae70-109731d63381" (UID: "be677a1a-5dc5-408e-ae70-109731d63381"). InnerVolumeSpecName "kube-api-access-5txhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:00:15.068397 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:15.068290    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be677a1a-5dc5-408e-ae70-109731d63381-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "be677a1a-5dc5-408e-ae70-109731d63381" (UID: "be677a1a-5dc5-408e-ae70-109731d63381"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:00:15.167619 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:15.167511    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be677a1a-5dc5-408e-ae70-109731d63381-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 19:00:15.167619 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:15.167561    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be677a1a-5dc5-408e-ae70-109731d63381-dshm\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 19:00:15.167619 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:15.167571    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be677a1a-5dc5-408e-ae70-109731d63381-model-cache\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 19:00:15.167619 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:15.167580    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5txhc\" (UniqueName: \"kubernetes.io/projected/be677a1a-5dc5-408e-ae70-109731d63381-kube-api-access-5txhc\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 19:00:15.922709 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:15.922662    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w" event={"ID":"be677a1a-5dc5-408e-ae70-109731d63381","Type":"ContainerDied","Data":"97765e5d86e5a84b4e058f9a7c012796310f7ec12e26669938f3a606f819ef5f"}
Sep 17 19:00:15.923168 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:15.922733    2581 scope.go:117] "RemoveContainer" containerID="24ce45c470826e6afb09141ac16753296b0c691079cb7996ac8645c5ea942438"
Sep 17 19:00:15.923168 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:15.922684    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w"
Sep 17 19:00:15.944823 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:15.944791    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w"]
Sep 17 19:00:15.948305 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:15.948276    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-cm55w"]
Sep 17 19:00:16.358153 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:16.358118    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be677a1a-5dc5-408e-ae70-109731d63381" path="/var/lib/kubelet/pods/be677a1a-5dc5-408e-ae70-109731d63381/volumes"
Sep 17 19:00:24.045831 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.045792    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"]
Sep 17 19:00:24.046374 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.046351    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be677a1a-5dc5-408e-ae70-109731d63381" containerName="main"
Sep 17 19:00:24.046457 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.046377    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="be677a1a-5dc5-408e-ae70-109731d63381" containerName="main"
Sep 17 19:00:24.046514 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.046486    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="be677a1a-5dc5-408e-ae70-109731d63381" containerName="main"
Sep 17 19:00:24.050977 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.050953    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"
Sep 17 19:00:24.053707 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.053680    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 17 19:00:24.053837 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.053772    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:00:24.061946 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.061921    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"]
Sep 17 19:00:24.147480 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.147445    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18ee3336-d4c0-46a9-81f0-4d6340eb4709-dshm\") pod \"prestop-hook-test-kserve-59999d87f8-rqpsb\" (UID: \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"
Sep 17 19:00:24.147632 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.147489    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18ee3336-d4c0-46a9-81f0-4d6340eb4709-home\") pod \"prestop-hook-test-kserve-59999d87f8-rqpsb\" (UID: \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"
Sep 17 19:00:24.147632 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.147548    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18ee3336-d4c0-46a9-81f0-4d6340eb4709-kserve-provision-location\") pod \"prestop-hook-test-kserve-59999d87f8-rqpsb\" (UID: \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"
Sep 17 19:00:24.147632 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.147572    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18ee3336-d4c0-46a9-81f0-4d6340eb4709-model-cache\") pod \"prestop-hook-test-kserve-59999d87f8-rqpsb\" (UID: \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"
Sep 17 19:00:24.147632 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.147611    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18ee3336-d4c0-46a9-81f0-4d6340eb4709-tmp-dir\") pod \"prestop-hook-test-kserve-59999d87f8-rqpsb\" (UID: \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"
Sep 17 19:00:24.147791 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.147672    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18ee3336-d4c0-46a9-81f0-4d6340eb4709-cabundle-cert\") pod \"prestop-hook-test-kserve-59999d87f8-rqpsb\" (UID: \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"
Sep 17 19:00:24.147791 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.147704    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18ee3336-d4c0-46a9-81f0-4d6340eb4709-tls-certs\") pod \"prestop-hook-test-kserve-59999d87f8-rqpsb\" (UID: \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"
Sep 17 19:00:24.147791 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.147725    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxqww\" (UniqueName: \"kubernetes.io/projected/18ee3336-d4c0-46a9-81f0-4d6340eb4709-kube-api-access-cxqww\") pod \"prestop-hook-test-kserve-59999d87f8-rqpsb\" (UID: \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"
Sep 17 19:00:24.248317 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.248277    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18ee3336-d4c0-46a9-81f0-4d6340eb4709-kserve-provision-location\") pod \"prestop-hook-test-kserve-59999d87f8-rqpsb\" (UID: \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"
Sep 17 19:00:24.248493 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.248326    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18ee3336-d4c0-46a9-81f0-4d6340eb4709-model-cache\") pod \"prestop-hook-test-kserve-59999d87f8-rqpsb\" (UID: \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"
Sep 17 19:00:24.248493 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.248366    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18ee3336-d4c0-46a9-81f0-4d6340eb4709-tmp-dir\") pod \"prestop-hook-test-kserve-59999d87f8-rqpsb\" (UID: \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"
Sep 17 19:00:24.248493 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.248386    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18ee3336-d4c0-46a9-81f0-4d6340eb4709-cabundle-cert\") pod \"prestop-hook-test-kserve-59999d87f8-rqpsb\" (UID: \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"
Sep 17 19:00:24.248493 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.248405    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18ee3336-d4c0-46a9-81f0-4d6340eb4709-tls-certs\") pod \"prestop-hook-test-kserve-59999d87f8-rqpsb\" (UID: \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"
Sep 17 19:00:24.248714 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.248571    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxqww\" (UniqueName: \"kubernetes.io/projected/18ee3336-d4c0-46a9-81f0-4d6340eb4709-kube-api-access-cxqww\") pod \"prestop-hook-test-kserve-59999d87f8-rqpsb\" (UID: \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"
Sep 17 19:00:24.248714 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.248623    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18ee3336-d4c0-46a9-81f0-4d6340eb4709-dshm\") pod \"prestop-hook-test-kserve-59999d87f8-rqpsb\" (UID: \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"
Sep 17 19:00:24.248714 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.248670    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18ee3336-d4c0-46a9-81f0-4d6340eb4709-home\") pod \"prestop-hook-test-kserve-59999d87f8-rqpsb\" (UID: \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"
Sep 17 19:00:24.248877 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.248738    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18ee3336-d4c0-46a9-81f0-4d6340eb4709-kserve-provision-location\") pod \"prestop-hook-test-kserve-59999d87f8-rqpsb\" (UID: \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"
Sep 17 19:00:24.248877 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.248782    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18ee3336-d4c0-46a9-81f0-4d6340eb4709-tmp-dir\") pod \"prestop-hook-test-kserve-59999d87f8-rqpsb\" (UID: \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"
Sep 17 19:00:24.249025 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.248936    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18ee3336-d4c0-46a9-81f0-4d6340eb4709-model-cache\") pod \"prestop-hook-test-kserve-59999d87f8-rqpsb\" (UID: \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"
Sep 17 19:00:24.249066 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.249044    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18ee3336-d4c0-46a9-81f0-4d6340eb4709-home\") pod \"prestop-hook-test-kserve-59999d87f8-rqpsb\" (UID: \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"
Sep 17 19:00:24.249201 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.249179    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18ee3336-d4c0-46a9-81f0-4d6340eb4709-cabundle-cert\") pod \"prestop-hook-test-kserve-59999d87f8-rqpsb\" (UID: \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"
Sep 17 19:00:24.250922 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.250887    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18ee3336-d4c0-46a9-81f0-4d6340eb4709-dshm\") pod \"prestop-hook-test-kserve-59999d87f8-rqpsb\" (UID: \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"
Sep 17 19:00:24.251068 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.251046    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18ee3336-d4c0-46a9-81f0-4d6340eb4709-tls-certs\") pod \"prestop-hook-test-kserve-59999d87f8-rqpsb\" (UID: \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"
Sep 17 19:00:24.257082 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.257064    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxqww\" (UniqueName: \"kubernetes.io/projected/18ee3336-d4c0-46a9-81f0-4d6340eb4709-kube-api-access-cxqww\") pod \"prestop-hook-test-kserve-59999d87f8-rqpsb\" (UID: \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"
Sep 17 19:00:24.361804 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.361718    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"
Sep 17 19:00:24.640046 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.640019    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"]
Sep 17 19:00:24.642218 ip-10-0-130-28 kubenswrapper[2581]: W0917 19:00:24.642186    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ee3336_d4c0_46a9_81f0_4d6340eb4709.slice/crio-c8c6269da00cc4c5e1d360babab79f9ab22f70d5e026bf9e9b1bb97da02acac5 WatchSource:0}: Error finding container c8c6269da00cc4c5e1d360babab79f9ab22f70d5e026bf9e9b1bb97da02acac5: Status 404 returned error can't find the container with id c8c6269da00cc4c5e1d360babab79f9ab22f70d5e026bf9e9b1bb97da02acac5
Sep 17 19:00:24.967046 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.967003    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb" event={"ID":"18ee3336-d4c0-46a9-81f0-4d6340eb4709","Type":"ContainerStarted","Data":"b8b14fb633b801aff1a9df7a2699fe8de3630232e7dae289f005ee4f0a9f9596"}
Sep 17 19:00:24.967252 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:24.967078    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb" event={"ID":"18ee3336-d4c0-46a9-81f0-4d6340eb4709","Type":"ContainerStarted","Data":"c8c6269da00cc4c5e1d360babab79f9ab22f70d5e026bf9e9b1bb97da02acac5"}
Sep 17 19:00:26.977773 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:26.977735    2581 generic.go:356] "Generic (PLEG): container finished" podID="18ee3336-d4c0-46a9-81f0-4d6340eb4709" containerID="b8b14fb633b801aff1a9df7a2699fe8de3630232e7dae289f005ee4f0a9f9596" exitCode=0
Sep 17 19:00:26.978204 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:26.977780    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb" event={"ID":"18ee3336-d4c0-46a9-81f0-4d6340eb4709","Type":"ContainerDied","Data":"b8b14fb633b801aff1a9df7a2699fe8de3630232e7dae289f005ee4f0a9f9596"}
Sep 17 19:00:27.984697 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:27.984662    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb" event={"ID":"18ee3336-d4c0-46a9-81f0-4d6340eb4709","Type":"ContainerStarted","Data":"f0603775ce660f7ec57952c2eb9115dc582f51c963c9f2d9810fb372eedf5a8a"}
Sep 17 19:00:28.008868 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:28.008812    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb" podStartSLOduration=1.761854516 podStartE2EDuration="4.00879522s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.246940704s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:00:28.006180186 +0000 UTC m=+3816.096801861" watchObservedRunningTime="2026-09-17 19:00:28.00879522 +0000 UTC m=+3816.099416869"
Sep 17 19:00:28.362344 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:28.362266    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"
Sep 17 19:00:34.362201 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:34.362166    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"
Sep 17 19:00:34.363681 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:34.363645    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb" podUID="18ee3336-d4c0-46a9-81f0-4d6340eb4709" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": dial tcp 10.133.0.81:8000: connect: connection refused"
Sep 17 19:00:44.362848 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:44.362797    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb" podUID="18ee3336-d4c0-46a9-81f0-4d6340eb4709" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": dial tcp 10.133.0.81:8000: connect: connection refused"
Sep 17 19:00:52.577410 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:52.577371    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"]
Sep 17 19:00:52.582545 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:52.582510    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp"]
Sep 17 19:00:54.370263 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:00:54.370214    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp" podUID="1ea8c903-de2a-4586-87f6-4faf70e702a6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:01:04.370350 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:04.370312    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp" podUID="1ea8c903-de2a-4586-87f6-4faf70e702a6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:01:07.606603 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:07.606552    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp" podUID="1ea8c903-de2a-4586-87f6-4faf70e702a6" containerName="main" containerID="cri-o://248160cf193f9854112e1298d04d57aa33cfc95b1fc373043568c1bd140492b5" gracePeriod=45
Sep 17 19:01:07.607044 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:07.606683    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb" podUID="18ee3336-d4c0-46a9-81f0-4d6340eb4709" containerName="main" containerID="cri-o://f0603775ce660f7ec57952c2eb9115dc582f51c963c9f2d9810fb372eedf5a8a" gracePeriod=45
Sep 17 19:01:09.543432 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:09.543403    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-59999d87f8-rqpsb_18ee3336-d4c0-46a9-81f0-4d6340eb4709/main/0.log"
Sep 17 19:01:09.544980 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:09.544960    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"
Sep 17 19:01:09.681809 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:09.681718    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18ee3336-d4c0-46a9-81f0-4d6340eb4709-kserve-provision-location\") pod \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\" (UID: \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\") "
Sep 17 19:01:09.681809 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:09.681767    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18ee3336-d4c0-46a9-81f0-4d6340eb4709-cabundle-cert\") pod \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\" (UID: \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\") "
Sep 17 19:01:09.681809 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:09.681787    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxqww\" (UniqueName: \"kubernetes.io/projected/18ee3336-d4c0-46a9-81f0-4d6340eb4709-kube-api-access-cxqww\") pod \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\" (UID: \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\") "
Sep 17 19:01:09.682145 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:09.681819    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18ee3336-d4c0-46a9-81f0-4d6340eb4709-home\") pod \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\" (UID: \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\") "
Sep 17 19:01:09.682145 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:09.681875    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18ee3336-d4c0-46a9-81f0-4d6340eb4709-model-cache\") pod \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\" (UID: \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\") "
Sep 17 19:01:09.682145 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:09.681925    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18ee3336-d4c0-46a9-81f0-4d6340eb4709-tmp-dir\") pod \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\" (UID: \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\") "
Sep 17 19:01:09.682145 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:09.681964    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18ee3336-d4c0-46a9-81f0-4d6340eb4709-dshm\") pod \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\" (UID: \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\") "
Sep 17 19:01:09.682145 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:09.682031    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18ee3336-d4c0-46a9-81f0-4d6340eb4709-tls-certs\") pod \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\" (UID: \"18ee3336-d4c0-46a9-81f0-4d6340eb4709\") "
Sep 17 19:01:09.682382 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:09.682183    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ee3336-d4c0-46a9-81f0-4d6340eb4709-model-cache" (OuterVolumeSpecName: "model-cache") pod "18ee3336-d4c0-46a9-81f0-4d6340eb4709" (UID: "18ee3336-d4c0-46a9-81f0-4d6340eb4709"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:01:09.682382 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:09.682213    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ee3336-d4c0-46a9-81f0-4d6340eb4709-home" (OuterVolumeSpecName: "home") pod "18ee3336-d4c0-46a9-81f0-4d6340eb4709" (UID: "18ee3336-d4c0-46a9-81f0-4d6340eb4709"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:01:09.682382 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:09.682252    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18ee3336-d4c0-46a9-81f0-4d6340eb4709-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "18ee3336-d4c0-46a9-81f0-4d6340eb4709" (UID: "18ee3336-d4c0-46a9-81f0-4d6340eb4709"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:01:09.682537 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:09.682395    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18ee3336-d4c0-46a9-81f0-4d6340eb4709-cabundle-cert\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 19:01:09.682537 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:09.682414    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18ee3336-d4c0-46a9-81f0-4d6340eb4709-home\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 19:01:09.682537 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:09.682427    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18ee3336-d4c0-46a9-81f0-4d6340eb4709-model-cache\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 19:01:09.682697 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:09.682550    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ee3336-d4c0-46a9-81f0-4d6340eb4709-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "18ee3336-d4c0-46a9-81f0-4d6340eb4709" (UID: "18ee3336-d4c0-46a9-81f0-4d6340eb4709"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:01:09.684360 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:09.684331    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ee3336-d4c0-46a9-81f0-4d6340eb4709-dshm" (OuterVolumeSpecName: "dshm") pod "18ee3336-d4c0-46a9-81f0-4d6340eb4709" (UID: "18ee3336-d4c0-46a9-81f0-4d6340eb4709"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:01:09.684473 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:09.684358    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ee3336-d4c0-46a9-81f0-4d6340eb4709-kube-api-access-cxqww" (OuterVolumeSpecName: "kube-api-access-cxqww") pod "18ee3336-d4c0-46a9-81f0-4d6340eb4709" (UID: "18ee3336-d4c0-46a9-81f0-4d6340eb4709"). InnerVolumeSpecName "kube-api-access-cxqww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:01:09.684473 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:09.684362    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18ee3336-d4c0-46a9-81f0-4d6340eb4709-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "18ee3336-d4c0-46a9-81f0-4d6340eb4709" (UID: "18ee3336-d4c0-46a9-81f0-4d6340eb4709"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:01:09.704507 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:09.704474    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ee3336-d4c0-46a9-81f0-4d6340eb4709-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "18ee3336-d4c0-46a9-81f0-4d6340eb4709" (UID: "18ee3336-d4c0-46a9-81f0-4d6340eb4709"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:01:09.784080 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:09.784035    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18ee3336-d4c0-46a9-81f0-4d6340eb4709-kserve-provision-location\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 19:01:09.784080 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:09.784077    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxqww\" (UniqueName: \"kubernetes.io/projected/18ee3336-d4c0-46a9-81f0-4d6340eb4709-kube-api-access-cxqww\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 19:01:09.784295 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:09.784108    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18ee3336-d4c0-46a9-81f0-4d6340eb4709-tmp-dir\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 19:01:09.784295 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:09.784117    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18ee3336-d4c0-46a9-81f0-4d6340eb4709-dshm\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 19:01:09.784295 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:09.784130    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18ee3336-d4c0-46a9-81f0-4d6340eb4709-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 19:01:10.199674 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:10.199641    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-59999d87f8-rqpsb_18ee3336-d4c0-46a9-81f0-4d6340eb4709/main/0.log"
Sep 17 19:01:10.200080 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:10.200056    2581 generic.go:356] "Generic (PLEG): container finished" podID="18ee3336-d4c0-46a9-81f0-4d6340eb4709" containerID="f0603775ce660f7ec57952c2eb9115dc582f51c963c9f2d9810fb372eedf5a8a" exitCode=130
Sep 17 19:01:10.200160 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:10.200108    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"
Sep 17 19:01:10.200200 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:10.200152    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb" event={"ID":"18ee3336-d4c0-46a9-81f0-4d6340eb4709","Type":"ContainerDied","Data":"f0603775ce660f7ec57952c2eb9115dc582f51c963c9f2d9810fb372eedf5a8a"}
Sep 17 19:01:10.200257 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:10.200242    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb" event={"ID":"18ee3336-d4c0-46a9-81f0-4d6340eb4709","Type":"ContainerDied","Data":"c8c6269da00cc4c5e1d360babab79f9ab22f70d5e026bf9e9b1bb97da02acac5"}
Sep 17 19:01:10.200296 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:10.200272    2581 scope.go:117] "RemoveContainer" containerID="f0603775ce660f7ec57952c2eb9115dc582f51c963c9f2d9810fb372eedf5a8a"
Sep 17 19:01:10.209614 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:10.209596    2581 scope.go:117] "RemoveContainer" containerID="b8b14fb633b801aff1a9df7a2699fe8de3630232e7dae289f005ee4f0a9f9596"
Sep 17 19:01:10.217555 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:10.217532    2581 scope.go:117] "RemoveContainer" containerID="f0603775ce660f7ec57952c2eb9115dc582f51c963c9f2d9810fb372eedf5a8a"
Sep 17 19:01:10.217853 ip-10-0-130-28 kubenswrapper[2581]: E0917 19:01:10.217834    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0603775ce660f7ec57952c2eb9115dc582f51c963c9f2d9810fb372eedf5a8a\": container with ID starting with f0603775ce660f7ec57952c2eb9115dc582f51c963c9f2d9810fb372eedf5a8a not found: ID does not exist" containerID="f0603775ce660f7ec57952c2eb9115dc582f51c963c9f2d9810fb372eedf5a8a"
Sep 17 19:01:10.217958 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:10.217862    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0603775ce660f7ec57952c2eb9115dc582f51c963c9f2d9810fb372eedf5a8a"} err="failed to get container status \"f0603775ce660f7ec57952c2eb9115dc582f51c963c9f2d9810fb372eedf5a8a\": rpc error: code = NotFound desc = could not find container \"f0603775ce660f7ec57952c2eb9115dc582f51c963c9f2d9810fb372eedf5a8a\": container with ID starting with f0603775ce660f7ec57952c2eb9115dc582f51c963c9f2d9810fb372eedf5a8a not found: ID does not exist"
Sep 17 19:01:10.217958 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:10.217880    2581 scope.go:117] "RemoveContainer" containerID="b8b14fb633b801aff1a9df7a2699fe8de3630232e7dae289f005ee4f0a9f9596"
Sep 17 19:01:10.218134 ip-10-0-130-28 kubenswrapper[2581]: E0917 19:01:10.218116    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8b14fb633b801aff1a9df7a2699fe8de3630232e7dae289f005ee4f0a9f9596\": container with ID starting with b8b14fb633b801aff1a9df7a2699fe8de3630232e7dae289f005ee4f0a9f9596 not found: ID does not exist" containerID="b8b14fb633b801aff1a9df7a2699fe8de3630232e7dae289f005ee4f0a9f9596"
Sep 17 19:01:10.218184 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:10.218141    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8b14fb633b801aff1a9df7a2699fe8de3630232e7dae289f005ee4f0a9f9596"} err="failed to get container status \"b8b14fb633b801aff1a9df7a2699fe8de3630232e7dae289f005ee4f0a9f9596\": rpc error: code = NotFound desc = could not find container \"b8b14fb633b801aff1a9df7a2699fe8de3630232e7dae289f005ee4f0a9f9596\": container with ID starting with b8b14fb633b801aff1a9df7a2699fe8de3630232e7dae289f005ee4f0a9f9596 not found: ID does not exist"
Sep 17 19:01:10.223962 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:10.223940    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"]
Sep 17 19:01:10.226258 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:10.226235    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-59999d87f8-rqpsb"]
Sep 17 19:01:10.358232 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:10.358197    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18ee3336-d4c0-46a9-81f0-4d6340eb4709" path="/var/lib/kubelet/pods/18ee3336-d4c0-46a9-81f0-4d6340eb4709/volumes"
Sep 17 19:01:14.370447 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:14.370409    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp" podUID="1ea8c903-de2a-4586-87f6-4faf70e702a6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:01:14.370868 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:14.370516    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp"
Sep 17 19:01:23.646295 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:23.646202    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps"]
Sep 17 19:01:23.646922 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:23.646726    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18ee3336-d4c0-46a9-81f0-4d6340eb4709" containerName="storage-initializer"
Sep 17 19:01:23.646922 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:23.646748    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="18ee3336-d4c0-46a9-81f0-4d6340eb4709" containerName="storage-initializer"
Sep 17 19:01:23.646922 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:23.646770    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18ee3336-d4c0-46a9-81f0-4d6340eb4709" containerName="main"
Sep 17 19:01:23.646922 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:23.646779    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="18ee3336-d4c0-46a9-81f0-4d6340eb4709" containerName="main"
Sep 17 19:01:23.646922 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:23.646892    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="18ee3336-d4c0-46a9-81f0-4d6340eb4709" containerName="main"
Sep 17 19:01:23.649499 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:23.649476    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps"
Sep 17 19:01:23.653867 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:23.653845    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 17 19:01:23.655122 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:23.655100    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 17 19:01:23.655228 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:23.655169    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-q8s6v\""
Sep 17 19:01:23.655298 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:23.655223    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-6xcb7\""
Sep 17 19:01:23.664027 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:23.664007    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 17 19:01:23.687116 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:23.687087    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps"]
Sep 17 19:01:23.805545 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:23.805503    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwdg5\" (UniqueName: \"kubernetes.io/projected/92db8fc7-e127-47b3-8712-8e7fc4967192-kube-api-access-jwdg5\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps\" (UID: \"92db8fc7-e127-47b3-8712-8e7fc4967192\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps"
Sep 17 19:01:23.805754 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:23.805557    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92db8fc7-e127-47b3-8712-8e7fc4967192-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps\" (UID: \"92db8fc7-e127-47b3-8712-8e7fc4967192\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps"
Sep 17 19:01:23.906881 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:23.906756    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwdg5\" (UniqueName: \"kubernetes.io/projected/92db8fc7-e127-47b3-8712-8e7fc4967192-kube-api-access-jwdg5\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps\" (UID: \"92db8fc7-e127-47b3-8712-8e7fc4967192\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps"
Sep 17 19:01:23.906881 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:23.906840    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92db8fc7-e127-47b3-8712-8e7fc4967192-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps\" (UID: \"92db8fc7-e127-47b3-8712-8e7fc4967192\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps"
Sep 17 19:01:23.909486 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:23.909460    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92db8fc7-e127-47b3-8712-8e7fc4967192-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps\" (UID: \"92db8fc7-e127-47b3-8712-8e7fc4967192\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps"
Sep 17 19:01:23.919699 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:23.919669    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwdg5\" (UniqueName: \"kubernetes.io/projected/92db8fc7-e127-47b3-8712-8e7fc4967192-kube-api-access-jwdg5\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps\" (UID: \"92db8fc7-e127-47b3-8712-8e7fc4967192\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps"
Sep 17 19:01:23.959604 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:23.959560    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps"
Sep 17 19:01:24.235957 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:24.235918    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps"]
Sep 17 19:01:24.241222 ip-10-0-130-28 kubenswrapper[2581]: W0917 19:01:24.241186    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92db8fc7_e127_47b3_8712_8e7fc4967192.slice/crio-65bc46520985c954eac0539bab37495cc8b1cbc727724c634d1e19d1c97eb6e9 WatchSource:0}: Error finding container 65bc46520985c954eac0539bab37495cc8b1cbc727724c634d1e19d1c97eb6e9: Status 404 returned error can't find the container with id 65bc46520985c954eac0539bab37495cc8b1cbc727724c634d1e19d1c97eb6e9
Sep 17 19:01:24.268553 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:24.268508    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps" event={"ID":"92db8fc7-e127-47b3-8712-8e7fc4967192","Type":"ContainerStarted","Data":"65bc46520985c954eac0539bab37495cc8b1cbc727724c634d1e19d1c97eb6e9"}
Sep 17 19:01:24.370779 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:24.370741    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp" podUID="1ea8c903-de2a-4586-87f6-4faf70e702a6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:01:25.275056 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:25.275017    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps" event={"ID":"92db8fc7-e127-47b3-8712-8e7fc4967192","Type":"ContainerStarted","Data":"d8302c3d2d450d8a788a547cc72927b8f8559104c163c7b73ce81f8e135445b3"}
Sep 17 19:01:25.275464 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:25.275153    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps"
Sep 17 19:01:25.295494 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:25.295441    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps" podStartSLOduration=2.295425357 podStartE2EDuration="2.295425357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:01:25.292732727 +0000 UTC m=+3873.383354403" watchObservedRunningTime="2026-09-17 19:01:25.295425357 +0000 UTC m=+3873.386047006"
Sep 17 19:01:34.370565 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:34.370523    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp" podUID="1ea8c903-de2a-4586-87f6-4faf70e702a6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:01:37.856083 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:37.856031    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp"
Sep 17 19:01:38.029766 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:38.029728    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7zwv\" (UniqueName: \"kubernetes.io/projected/1ea8c903-de2a-4586-87f6-4faf70e702a6-kube-api-access-v7zwv\") pod \"1ea8c903-de2a-4586-87f6-4faf70e702a6\" (UID: \"1ea8c903-de2a-4586-87f6-4faf70e702a6\") "
Sep 17 19:01:38.029979 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:38.029810    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ea8c903-de2a-4586-87f6-4faf70e702a6-tls-certs\") pod \"1ea8c903-de2a-4586-87f6-4faf70e702a6\" (UID: \"1ea8c903-de2a-4586-87f6-4faf70e702a6\") "
Sep 17 19:01:38.031946 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:38.031914    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ea8c903-de2a-4586-87f6-4faf70e702a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1ea8c903-de2a-4586-87f6-4faf70e702a6" (UID: "1ea8c903-de2a-4586-87f6-4faf70e702a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:01:38.032069 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:38.031952    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ea8c903-de2a-4586-87f6-4faf70e702a6-kube-api-access-v7zwv" (OuterVolumeSpecName: "kube-api-access-v7zwv") pod "1ea8c903-de2a-4586-87f6-4faf70e702a6" (UID: "1ea8c903-de2a-4586-87f6-4faf70e702a6"). InnerVolumeSpecName "kube-api-access-v7zwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:01:38.131153 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:38.131114    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7zwv\" (UniqueName: \"kubernetes.io/projected/1ea8c903-de2a-4586-87f6-4faf70e702a6-kube-api-access-v7zwv\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 19:01:38.131153 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:38.131153    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ea8c903-de2a-4586-87f6-4faf70e702a6-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 19:01:38.335018 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:38.334922    2581 generic.go:356] "Generic (PLEG): container finished" podID="1ea8c903-de2a-4586-87f6-4faf70e702a6" containerID="248160cf193f9854112e1298d04d57aa33cfc95b1fc373043568c1bd140492b5" exitCode=0
Sep 17 19:01:38.335018 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:38.334966    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp"
Sep 17 19:01:38.335219 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:38.335007    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp" event={"ID":"1ea8c903-de2a-4586-87f6-4faf70e702a6","Type":"ContainerDied","Data":"248160cf193f9854112e1298d04d57aa33cfc95b1fc373043568c1bd140492b5"}
Sep 17 19:01:38.335219 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:38.335069    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp" event={"ID":"1ea8c903-de2a-4586-87f6-4faf70e702a6","Type":"ContainerDied","Data":"dc6903d7f789368e4e0de5bea5b6785f78dd1e0e0a855470149f134f0fe59a00"}
Sep 17 19:01:38.335219 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:38.335097    2581 scope.go:117] "RemoveContainer" containerID="248160cf193f9854112e1298d04d57aa33cfc95b1fc373043568c1bd140492b5"
Sep 17 19:01:38.344161 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:38.344144    2581 scope.go:117] "RemoveContainer" containerID="248160cf193f9854112e1298d04d57aa33cfc95b1fc373043568c1bd140492b5"
Sep 17 19:01:38.344447 ip-10-0-130-28 kubenswrapper[2581]: E0917 19:01:38.344425    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"248160cf193f9854112e1298d04d57aa33cfc95b1fc373043568c1bd140492b5\": container with ID starting with 248160cf193f9854112e1298d04d57aa33cfc95b1fc373043568c1bd140492b5 not found: ID does not exist" containerID="248160cf193f9854112e1298d04d57aa33cfc95b1fc373043568c1bd140492b5"
Sep 17 19:01:38.344517 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:38.344455    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"248160cf193f9854112e1298d04d57aa33cfc95b1fc373043568c1bd140492b5"} err="failed to get container status \"248160cf193f9854112e1298d04d57aa33cfc95b1fc373043568c1bd140492b5\": rpc error: code = NotFound desc = could not find container \"248160cf193f9854112e1298d04d57aa33cfc95b1fc373043568c1bd140492b5\": container with ID starting with 248160cf193f9854112e1298d04d57aa33cfc95b1fc373043568c1bd140492b5 not found: ID does not exist"
Sep 17 19:01:38.359162 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:38.359129    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp"]
Sep 17 19:01:38.363261 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:38.363238    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-29dcp"]
Sep 17 19:01:40.358658 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:40.358611    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ea8c903-de2a-4586-87f6-4faf70e702a6" path="/var/lib/kubelet/pods/1ea8c903-de2a-4586-87f6-4faf70e702a6/volumes"
Sep 17 19:01:51.175595 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:51.175557    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk"]
Sep 17 19:01:51.175987 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:51.175967    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ea8c903-de2a-4586-87f6-4faf70e702a6" containerName="main"
Sep 17 19:01:51.175987 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:51.175980    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ea8c903-de2a-4586-87f6-4faf70e702a6" containerName="main"
Sep 17 19:01:51.176063 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:51.176053    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ea8c903-de2a-4586-87f6-4faf70e702a6" containerName="main"
Sep 17 19:01:51.178613 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:51.178594    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk"
Sep 17 19:01:51.182208 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:51.182186    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 17 19:01:51.182208 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:51.182195    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 17 19:01:51.182208 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:51.182187    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-8zwt9\""
Sep 17 19:01:51.182455 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:51.182200    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 17 19:01:51.191081 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:51.191058    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk"]
Sep 17 19:01:51.235465 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:51.235426    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04890050-74a1-4256-8347-b4c44e09c06e-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx4vk\" (UID: \"04890050-74a1-4256-8347-b4c44e09c06e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk"
Sep 17 19:01:51.235657 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:51.235481    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04890050-74a1-4256-8347-b4c44e09c06e-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx4vk\" (UID: \"04890050-74a1-4256-8347-b4c44e09c06e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk"
Sep 17 19:01:51.235657 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:51.235507    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04890050-74a1-4256-8347-b4c44e09c06e-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx4vk\" (UID: \"04890050-74a1-4256-8347-b4c44e09c06e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk"
Sep 17 19:01:51.235657 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:51.235536    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqjff\" (UniqueName: \"kubernetes.io/projected/04890050-74a1-4256-8347-b4c44e09c06e-kube-api-access-nqjff\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx4vk\" (UID: \"04890050-74a1-4256-8347-b4c44e09c06e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk"
Sep 17 19:01:51.235657 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:51.235561    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04890050-74a1-4256-8347-b4c44e09c06e-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx4vk\" (UID: \"04890050-74a1-4256-8347-b4c44e09c06e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk"
Sep 17 19:01:51.235657 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:51.235582    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04890050-74a1-4256-8347-b4c44e09c06e-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx4vk\" (UID: \"04890050-74a1-4256-8347-b4c44e09c06e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk"
Sep 17 19:01:51.336830 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:51.336787    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04890050-74a1-4256-8347-b4c44e09c06e-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx4vk\" (UID: \"04890050-74a1-4256-8347-b4c44e09c06e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk"
Sep 17 19:01:51.337062 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:51.336844    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04890050-74a1-4256-8347-b4c44e09c06e-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx4vk\" (UID: \"04890050-74a1-4256-8347-b4c44e09c06e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk"
Sep 17 19:01:51.337062 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:51.336869    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04890050-74a1-4256-8347-b4c44e09c06e-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx4vk\" (UID: \"04890050-74a1-4256-8347-b4c44e09c06e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk"
Sep 17 19:01:51.337062 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:51.336944    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqjff\" (UniqueName: \"kubernetes.io/projected/04890050-74a1-4256-8347-b4c44e09c06e-kube-api-access-nqjff\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx4vk\" (UID: \"04890050-74a1-4256-8347-b4c44e09c06e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk"
Sep 17 19:01:51.337062 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:51.336993    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04890050-74a1-4256-8347-b4c44e09c06e-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx4vk\" (UID: \"04890050-74a1-4256-8347-b4c44e09c06e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk"
Sep 17 19:01:51.337062 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:51.337018    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04890050-74a1-4256-8347-b4c44e09c06e-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx4vk\" (UID: \"04890050-74a1-4256-8347-b4c44e09c06e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk"
Sep 17 19:01:51.337338 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:51.337315    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04890050-74a1-4256-8347-b4c44e09c06e-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx4vk\" (UID: \"04890050-74a1-4256-8347-b4c44e09c06e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk"
Sep 17 19:01:51.337387 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:51.337359    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04890050-74a1-4256-8347-b4c44e09c06e-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx4vk\" (UID: \"04890050-74a1-4256-8347-b4c44e09c06e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk"
Sep 17 19:01:51.337456 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:51.337414    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04890050-74a1-4256-8347-b4c44e09c06e-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx4vk\" (UID: \"04890050-74a1-4256-8347-b4c44e09c06e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk"
Sep 17 19:01:51.339313 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:51.339288    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04890050-74a1-4256-8347-b4c44e09c06e-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx4vk\" (UID: \"04890050-74a1-4256-8347-b4c44e09c06e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk"
Sep 17 19:01:51.339500 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:51.339481    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04890050-74a1-4256-8347-b4c44e09c06e-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx4vk\" (UID: \"04890050-74a1-4256-8347-b4c44e09c06e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk"
Sep 17 19:01:51.345896 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:51.345863    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqjff\" (UniqueName: \"kubernetes.io/projected/04890050-74a1-4256-8347-b4c44e09c06e-kube-api-access-nqjff\") pod \"rolling-upgrade-test-kserve-74b79d7f77-gx4vk\" (UID: \"04890050-74a1-4256-8347-b4c44e09c06e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk"
Sep 17 19:01:51.488249 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:51.488203    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk"
Sep 17 19:01:51.761393 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:51.761366    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk"]
Sep 17 19:01:51.763516 ip-10-0-130-28 kubenswrapper[2581]: W0917 19:01:51.763486    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04890050_74a1_4256_8347_b4c44e09c06e.slice/crio-80775e57d7a1c1cd4d213a70626a3fbad81df7fc78b5e97973dac931bc41d66e WatchSource:0}: Error finding container 80775e57d7a1c1cd4d213a70626a3fbad81df7fc78b5e97973dac931bc41d66e: Status 404 returned error can't find the container with id 80775e57d7a1c1cd4d213a70626a3fbad81df7fc78b5e97973dac931bc41d66e
Sep 17 19:01:52.402133 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:52.402095    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk" event={"ID":"04890050-74a1-4256-8347-b4c44e09c06e","Type":"ContainerStarted","Data":"c2a8fbcc8930aab382b635bc7cd404b95ce124f0550b79db362a8f12e5d21b3c"}
Sep 17 19:01:52.402633 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:52.402152    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk" event={"ID":"04890050-74a1-4256-8347-b4c44e09c06e","Type":"ContainerStarted","Data":"80775e57d7a1c1cd4d213a70626a3fbad81df7fc78b5e97973dac931bc41d66e"}
Sep 17 19:01:52.421575 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:52.421517    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk" podStartSLOduration=1.421501034 podStartE2EDuration="1.421501034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:01:52.420671204 +0000 UTC m=+3900.511292853" watchObservedRunningTime="2026-09-17 19:01:52.421501034 +0000 UTC m=+3900.512122684"
Sep 17 19:01:52.488848 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:52.488810    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk"
Sep 17 19:01:52.705648 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:52.705622    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/2.log"
Sep 17 19:01:52.732682 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:52.732652    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/2.log"
Sep 17 19:01:56.281335 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:01:56.281304    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps"
Sep 17 19:02:01.489129 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:01.489081    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk"
Sep 17 19:02:01.501740 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:01.501708    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk"
Sep 17 19:02:02.461001 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:02.460968    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk"
Sep 17 19:02:08.220437 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:08.220400    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps"]
Sep 17 19:02:16.281695 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:16.281654    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps" podUID="92db8fc7-e127-47b3-8712-8e7fc4967192" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:02:23.244279 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:23.244233    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps" podUID="92db8fc7-e127-47b3-8712-8e7fc4967192" containerName="main" containerID="cri-o://d8302c3d2d450d8a788a547cc72927b8f8559104c163c7b73ce81f8e135445b3" gracePeriod=45
Sep 17 19:02:26.281331 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:26.281291    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps" podUID="92db8fc7-e127-47b3-8712-8e7fc4967192" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:02:36.167923 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:36.167867    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk"]
Sep 17 19:02:36.179982 ip-10-0-130-28 kubenswrapper[2581]: E0917 19:02:36.179941    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:02:36.179982 ip-10-0-130-28 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:02:36.179982 ip-10-0-130-28 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 19:02:36.179982 ip-10-0-130-28 kubenswrapper[2581]:  > containerID="c2a8fbcc8930aab382b635bc7cd404b95ce124f0550b79db362a8f12e5d21b3c" cmd=["/bin/sleep","15"]
Sep 17 19:02:36.180275 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:36.180002    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk" message="" err=<
Sep 17 19:02:36.180275 ip-10-0-130-28 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:02:36.180275 ip-10-0-130-28 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 19:02:36.180275 ip-10-0-130-28 kubenswrapper[2581]:  >
Sep 17 19:02:36.180275 ip-10-0-130-28 kubenswrapper[2581]: E0917 19:02:36.180016    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:02:36.180275 ip-10-0-130-28 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:02:36.180275 ip-10-0-130-28 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 19:02:36.180275 ip-10-0-130-28 kubenswrapper[2581]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk" podUID="04890050-74a1-4256-8347-b4c44e09c06e" containerName="main" containerID="cri-o://c2a8fbcc8930aab382b635bc7cd404b95ce124f0550b79db362a8f12e5d21b3c"
Sep 17 19:02:36.180275 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:36.180054    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk" podUID="04890050-74a1-4256-8347-b4c44e09c06e" containerName="main" containerID="cri-o://c2a8fbcc8930aab382b635bc7cd404b95ce124f0550b79db362a8f12e5d21b3c" gracePeriod=60
Sep 17 19:02:36.280826 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:36.280783    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps" podUID="92db8fc7-e127-47b3-8712-8e7fc4967192" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:02:36.281027 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:36.280941    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps"
Sep 17 19:02:36.520741 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:36.520717    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk"
Sep 17 19:02:36.554787 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:36.554758    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04890050-74a1-4256-8347-b4c44e09c06e-model-cache\") pod \"04890050-74a1-4256-8347-b4c44e09c06e\" (UID: \"04890050-74a1-4256-8347-b4c44e09c06e\") "
Sep 17 19:02:36.554958 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:36.554798    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04890050-74a1-4256-8347-b4c44e09c06e-tls-certs\") pod \"04890050-74a1-4256-8347-b4c44e09c06e\" (UID: \"04890050-74a1-4256-8347-b4c44e09c06e\") "
Sep 17 19:02:36.554958 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:36.554823    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04890050-74a1-4256-8347-b4c44e09c06e-tmp-dir\") pod \"04890050-74a1-4256-8347-b4c44e09c06e\" (UID: \"04890050-74a1-4256-8347-b4c44e09c06e\") "
Sep 17 19:02:36.554958 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:36.554852    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqjff\" (UniqueName: \"kubernetes.io/projected/04890050-74a1-4256-8347-b4c44e09c06e-kube-api-access-nqjff\") pod \"04890050-74a1-4256-8347-b4c44e09c06e\" (UID: \"04890050-74a1-4256-8347-b4c44e09c06e\") "
Sep 17 19:02:36.554958 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:36.554878    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04890050-74a1-4256-8347-b4c44e09c06e-home\") pod \"04890050-74a1-4256-8347-b4c44e09c06e\" (UID: \"04890050-74a1-4256-8347-b4c44e09c06e\") "
Sep 17 19:02:36.554958 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:36.554938    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04890050-74a1-4256-8347-b4c44e09c06e-dshm\") pod \"04890050-74a1-4256-8347-b4c44e09c06e\" (UID: \"04890050-74a1-4256-8347-b4c44e09c06e\") "
Sep 17 19:02:36.555257 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:36.555123    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04890050-74a1-4256-8347-b4c44e09c06e-model-cache" (OuterVolumeSpecName: "model-cache") pod "04890050-74a1-4256-8347-b4c44e09c06e" (UID: "04890050-74a1-4256-8347-b4c44e09c06e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:02:36.555257 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:36.555141    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04890050-74a1-4256-8347-b4c44e09c06e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "04890050-74a1-4256-8347-b4c44e09c06e" (UID: "04890050-74a1-4256-8347-b4c44e09c06e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:02:36.555257 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:36.555198    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04890050-74a1-4256-8347-b4c44e09c06e-home" (OuterVolumeSpecName: "home") pod "04890050-74a1-4256-8347-b4c44e09c06e" (UID: "04890050-74a1-4256-8347-b4c44e09c06e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:02:36.557162 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:36.557131    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04890050-74a1-4256-8347-b4c44e09c06e-dshm" (OuterVolumeSpecName: "dshm") pod "04890050-74a1-4256-8347-b4c44e09c06e" (UID: "04890050-74a1-4256-8347-b4c44e09c06e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:02:36.557298 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:36.557272    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04890050-74a1-4256-8347-b4c44e09c06e-kube-api-access-nqjff" (OuterVolumeSpecName: "kube-api-access-nqjff") pod "04890050-74a1-4256-8347-b4c44e09c06e" (UID: "04890050-74a1-4256-8347-b4c44e09c06e"). InnerVolumeSpecName "kube-api-access-nqjff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:02:36.557298 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:36.557292    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04890050-74a1-4256-8347-b4c44e09c06e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "04890050-74a1-4256-8347-b4c44e09c06e" (UID: "04890050-74a1-4256-8347-b4c44e09c06e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:02:36.605184 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:36.605147    2581 generic.go:356] "Generic (PLEG): container finished" podID="04890050-74a1-4256-8347-b4c44e09c06e" containerID="c2a8fbcc8930aab382b635bc7cd404b95ce124f0550b79db362a8f12e5d21b3c" exitCode=0
Sep 17 19:02:36.605316 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:36.605192    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk"
Sep 17 19:02:36.605316 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:36.605224    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk" event={"ID":"04890050-74a1-4256-8347-b4c44e09c06e","Type":"ContainerDied","Data":"c2a8fbcc8930aab382b635bc7cd404b95ce124f0550b79db362a8f12e5d21b3c"}
Sep 17 19:02:36.605316 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:36.605270    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk" event={"ID":"04890050-74a1-4256-8347-b4c44e09c06e","Type":"ContainerDied","Data":"80775e57d7a1c1cd4d213a70626a3fbad81df7fc78b5e97973dac931bc41d66e"}
Sep 17 19:02:36.605316 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:36.605294    2581 scope.go:117] "RemoveContainer" containerID="c2a8fbcc8930aab382b635bc7cd404b95ce124f0550b79db362a8f12e5d21b3c"
Sep 17 19:02:36.614592 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:36.614573    2581 scope.go:117] "RemoveContainer" containerID="c2a8fbcc8930aab382b635bc7cd404b95ce124f0550b79db362a8f12e5d21b3c"
Sep 17 19:02:36.614863 ip-10-0-130-28 kubenswrapper[2581]: E0917 19:02:36.614842    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2a8fbcc8930aab382b635bc7cd404b95ce124f0550b79db362a8f12e5d21b3c\": container with ID starting with c2a8fbcc8930aab382b635bc7cd404b95ce124f0550b79db362a8f12e5d21b3c not found: ID does not exist" containerID="c2a8fbcc8930aab382b635bc7cd404b95ce124f0550b79db362a8f12e5d21b3c"
Sep 17 19:02:36.614928 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:36.614871    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2a8fbcc8930aab382b635bc7cd404b95ce124f0550b79db362a8f12e5d21b3c"} err="failed to get container status \"c2a8fbcc8930aab382b635bc7cd404b95ce124f0550b79db362a8f12e5d21b3c\": rpc error: code = NotFound desc = could not find container \"c2a8fbcc8930aab382b635bc7cd404b95ce124f0550b79db362a8f12e5d21b3c\": container with ID starting with c2a8fbcc8930aab382b635bc7cd404b95ce124f0550b79db362a8f12e5d21b3c not found: ID does not exist"
Sep 17 19:02:36.636160 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:36.636136    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk"]
Sep 17 19:02:36.639195 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:36.639174    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk"]
Sep 17 19:02:36.656107 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:36.656087    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqjff\" (UniqueName: \"kubernetes.io/projected/04890050-74a1-4256-8347-b4c44e09c06e-kube-api-access-nqjff\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 19:02:36.656107 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:36.656107    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04890050-74a1-4256-8347-b4c44e09c06e-home\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 19:02:36.656288 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:36.656117    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04890050-74a1-4256-8347-b4c44e09c06e-dshm\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 19:02:36.656288 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:36.656125    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04890050-74a1-4256-8347-b4c44e09c06e-model-cache\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 19:02:36.656288 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:36.656134    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04890050-74a1-4256-8347-b4c44e09c06e-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 19:02:36.656288 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:36.656165    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04890050-74a1-4256-8347-b4c44e09c06e-tmp-dir\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 19:02:37.450867 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:37.450812    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-gx4vk" podUID="04890050-74a1-4256-8347-b4c44e09c06e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": context deadline exceeded"
Sep 17 19:02:38.359427 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:38.359394    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04890050-74a1-4256-8347-b4c44e09c06e" path="/var/lib/kubelet/pods/04890050-74a1-4256-8347-b4c44e09c06e/volumes"
Sep 17 19:02:46.281444 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:46.281348    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps" podUID="92db8fc7-e127-47b3-8712-8e7fc4967192" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:02:53.502400 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:53.502368    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps"
Sep 17 19:02:53.614972 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:53.614850    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwdg5\" (UniqueName: \"kubernetes.io/projected/92db8fc7-e127-47b3-8712-8e7fc4967192-kube-api-access-jwdg5\") pod \"92db8fc7-e127-47b3-8712-8e7fc4967192\" (UID: \"92db8fc7-e127-47b3-8712-8e7fc4967192\") "
Sep 17 19:02:53.614972 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:53.614952    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92db8fc7-e127-47b3-8712-8e7fc4967192-tls-certs\") pod \"92db8fc7-e127-47b3-8712-8e7fc4967192\" (UID: \"92db8fc7-e127-47b3-8712-8e7fc4967192\") "
Sep 17 19:02:53.617220 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:53.617186    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92db8fc7-e127-47b3-8712-8e7fc4967192-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "92db8fc7-e127-47b3-8712-8e7fc4967192" (UID: "92db8fc7-e127-47b3-8712-8e7fc4967192"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:02:53.617220 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:53.617187    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92db8fc7-e127-47b3-8712-8e7fc4967192-kube-api-access-jwdg5" (OuterVolumeSpecName: "kube-api-access-jwdg5") pod "92db8fc7-e127-47b3-8712-8e7fc4967192" (UID: "92db8fc7-e127-47b3-8712-8e7fc4967192"). InnerVolumeSpecName "kube-api-access-jwdg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:02:53.686544 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:53.686502    2581 generic.go:356] "Generic (PLEG): container finished" podID="92db8fc7-e127-47b3-8712-8e7fc4967192" containerID="d8302c3d2d450d8a788a547cc72927b8f8559104c163c7b73ce81f8e135445b3" exitCode=0
Sep 17 19:02:53.686544 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:53.686544    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps"
Sep 17 19:02:53.686747 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:53.686586    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps" event={"ID":"92db8fc7-e127-47b3-8712-8e7fc4967192","Type":"ContainerDied","Data":"d8302c3d2d450d8a788a547cc72927b8f8559104c163c7b73ce81f8e135445b3"}
Sep 17 19:02:53.686747 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:53.686632    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps" event={"ID":"92db8fc7-e127-47b3-8712-8e7fc4967192","Type":"ContainerDied","Data":"65bc46520985c954eac0539bab37495cc8b1cbc727724c634d1e19d1c97eb6e9"}
Sep 17 19:02:53.686747 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:53.686654    2581 scope.go:117] "RemoveContainer" containerID="d8302c3d2d450d8a788a547cc72927b8f8559104c163c7b73ce81f8e135445b3"
Sep 17 19:02:53.695660 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:53.695638    2581 scope.go:117] "RemoveContainer" containerID="d8302c3d2d450d8a788a547cc72927b8f8559104c163c7b73ce81f8e135445b3"
Sep 17 19:02:53.695958 ip-10-0-130-28 kubenswrapper[2581]: E0917 19:02:53.695938    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8302c3d2d450d8a788a547cc72927b8f8559104c163c7b73ce81f8e135445b3\": container with ID starting with d8302c3d2d450d8a788a547cc72927b8f8559104c163c7b73ce81f8e135445b3 not found: ID does not exist" containerID="d8302c3d2d450d8a788a547cc72927b8f8559104c163c7b73ce81f8e135445b3"
Sep 17 19:02:53.696020 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:53.695971    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8302c3d2d450d8a788a547cc72927b8f8559104c163c7b73ce81f8e135445b3"} err="failed to get container status \"d8302c3d2d450d8a788a547cc72927b8f8559104c163c7b73ce81f8e135445b3\": rpc error: code = NotFound desc = could not find container \"d8302c3d2d450d8a788a547cc72927b8f8559104c163c7b73ce81f8e135445b3\": container with ID starting with d8302c3d2d450d8a788a547cc72927b8f8559104c163c7b73ce81f8e135445b3 not found: ID does not exist"
Sep 17 19:02:53.716004 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:53.715960    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwdg5\" (UniqueName: \"kubernetes.io/projected/92db8fc7-e127-47b3-8712-8e7fc4967192-kube-api-access-jwdg5\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 19:02:53.716004 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:53.716001    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92db8fc7-e127-47b3-8712-8e7fc4967192-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 19:02:53.719584 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:53.719553    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps"]
Sep 17 19:02:53.724451 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:53.724420    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-scherb4ps"]
Sep 17 19:02:54.359195 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:02:54.359147    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92db8fc7-e127-47b3-8712-8e7fc4967192" path="/var/lib/kubelet/pods/92db8fc7-e127-47b3-8712-8e7fc4967192/volumes"
Sep 17 19:03:07.821322 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:07.821286    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s"]
Sep 17 19:03:07.821922 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:07.821642    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04890050-74a1-4256-8347-b4c44e09c06e" containerName="main"
Sep 17 19:03:07.821922 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:07.821653    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="04890050-74a1-4256-8347-b4c44e09c06e" containerName="main"
Sep 17 19:03:07.821922 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:07.821666    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92db8fc7-e127-47b3-8712-8e7fc4967192" containerName="main"
Sep 17 19:03:07.821922 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:07.821672    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="92db8fc7-e127-47b3-8712-8e7fc4967192" containerName="main"
Sep 17 19:03:07.821922 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:07.821739    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="92db8fc7-e127-47b3-8712-8e7fc4967192" containerName="main"
Sep 17 19:03:07.821922 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:07.821751    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="04890050-74a1-4256-8347-b4c44e09c06e" containerName="main"
Sep 17 19:03:07.826540 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:07.826516    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s"
Sep 17 19:03:07.834043 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:07.834022    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 17 19:03:07.835085 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:07.835065    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 17 19:03:07.835190 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:07.835088    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 17 19:03:07.835190 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:07.835090    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-8pqb7\""
Sep 17 19:03:07.850486 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:07.850455    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s"]
Sep 17 19:03:07.938312 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:07.938275    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s\" (UID: \"5fdd73d8-efc1-4dae-a841-48ab4f4f521c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s"
Sep 17 19:03:07.938312 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:07.938324    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj749\" (UniqueName: \"kubernetes.io/projected/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-kube-api-access-wj749\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s\" (UID: \"5fdd73d8-efc1-4dae-a841-48ab4f4f521c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s"
Sep 17 19:03:07.938543 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:07.938360    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s\" (UID: \"5fdd73d8-efc1-4dae-a841-48ab4f4f521c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s"
Sep 17 19:03:07.938543 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:07.938385    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s\" (UID: \"5fdd73d8-efc1-4dae-a841-48ab4f4f521c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s"
Sep 17 19:03:07.938543 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:07.938413    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s\" (UID: \"5fdd73d8-efc1-4dae-a841-48ab4f4f521c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s"
Sep 17 19:03:07.938543 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:07.938460    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s\" (UID: \"5fdd73d8-efc1-4dae-a841-48ab4f4f521c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s"
Sep 17 19:03:08.039458 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:08.039419    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s\" (UID: \"5fdd73d8-efc1-4dae-a841-48ab4f4f521c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s"
Sep 17 19:03:08.039458 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:08.039469    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s\" (UID: \"5fdd73d8-efc1-4dae-a841-48ab4f4f521c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s"
Sep 17 19:03:08.039713 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:08.039512    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s\" (UID: \"5fdd73d8-efc1-4dae-a841-48ab4f4f521c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s"
Sep 17 19:03:08.039713 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:08.039570    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s\" (UID: \"5fdd73d8-efc1-4dae-a841-48ab4f4f521c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s"
Sep 17 19:03:08.039713 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:08.039598    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s\" (UID: \"5fdd73d8-efc1-4dae-a841-48ab4f4f521c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s"
Sep 17 19:03:08.039713 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:08.039637    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj749\" (UniqueName: \"kubernetes.io/projected/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-kube-api-access-wj749\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s\" (UID: \"5fdd73d8-efc1-4dae-a841-48ab4f4f521c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s"
Sep 17 19:03:08.039964 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:08.039889    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s\" (UID: \"5fdd73d8-efc1-4dae-a841-48ab4f4f521c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s"
Sep 17 19:03:08.040039 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:08.040010    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s\" (UID: \"5fdd73d8-efc1-4dae-a841-48ab4f4f521c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s"
Sep 17 19:03:08.040098 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:08.040067    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s\" (UID: \"5fdd73d8-efc1-4dae-a841-48ab4f4f521c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s"
Sep 17 19:03:08.041870 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:08.041840    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s\" (UID: \"5fdd73d8-efc1-4dae-a841-48ab4f4f521c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s"
Sep 17 19:03:08.042092 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:08.042074    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s\" (UID: \"5fdd73d8-efc1-4dae-a841-48ab4f4f521c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s"
Sep 17 19:03:08.050728 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:08.050707    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj749\" (UniqueName: \"kubernetes.io/projected/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-kube-api-access-wj749\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s\" (UID: \"5fdd73d8-efc1-4dae-a841-48ab4f4f521c\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s"
Sep 17 19:03:08.124042 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:08.123953    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd"]
Sep 17 19:03:08.127645 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:08.127622    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd"
Sep 17 19:03:08.131035 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:08.131013    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-xp9ph\""
Sep 17 19:03:08.136422 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:08.136397    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s"
Sep 17 19:03:08.142092 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:08.142071    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd"]
Sep 17 19:03:08.241510 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:08.241469    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e999ce07-27aa-48c1-bddd-019e1c8fd731-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd\" (UID: \"e999ce07-27aa-48c1-bddd-019e1c8fd731\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd"
Sep 17 19:03:08.241510 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:08.241511    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n7df\" (UniqueName: \"kubernetes.io/projected/e999ce07-27aa-48c1-bddd-019e1c8fd731-kube-api-access-4n7df\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd\" (UID: \"e999ce07-27aa-48c1-bddd-019e1c8fd731\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd"
Sep 17 19:03:08.342247 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:08.342214    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e999ce07-27aa-48c1-bddd-019e1c8fd731-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd\" (UID: \"e999ce07-27aa-48c1-bddd-019e1c8fd731\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd"
Sep 17 19:03:08.342427 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:08.342252    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n7df\" (UniqueName: \"kubernetes.io/projected/e999ce07-27aa-48c1-bddd-019e1c8fd731-kube-api-access-4n7df\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd\" (UID: \"e999ce07-27aa-48c1-bddd-019e1c8fd731\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd"
Sep 17 19:03:08.344730 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:08.344698    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e999ce07-27aa-48c1-bddd-019e1c8fd731-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd\" (UID: \"e999ce07-27aa-48c1-bddd-019e1c8fd731\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd"
Sep 17 19:03:08.353885 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:08.353853    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n7df\" (UniqueName: \"kubernetes.io/projected/e999ce07-27aa-48c1-bddd-019e1c8fd731-kube-api-access-4n7df\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd\" (UID: \"e999ce07-27aa-48c1-bddd-019e1c8fd731\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd"
Sep 17 19:03:08.420753 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:08.420721    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s"]
Sep 17 19:03:08.422043 ip-10-0-130-28 kubenswrapper[2581]: W0917 19:03:08.422015    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fdd73d8_efc1_4dae_a841_48ab4f4f521c.slice/crio-04453d6ec12b0e2168dac9f8a94ed75e6cf9c9e4b24c40498bb71975241db553 WatchSource:0}: Error finding container 04453d6ec12b0e2168dac9f8a94ed75e6cf9c9e4b24c40498bb71975241db553: Status 404 returned error can't find the container with id 04453d6ec12b0e2168dac9f8a94ed75e6cf9c9e4b24c40498bb71975241db553
Sep 17 19:03:08.438102 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:08.438063    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd"
Sep 17 19:03:08.731396 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:08.731359    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd"]
Sep 17 19:03:08.734602 ip-10-0-130-28 kubenswrapper[2581]: W0917 19:03:08.734568    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode999ce07_27aa_48c1_bddd_019e1c8fd731.slice/crio-8a58fb29c96750a7fea062e286cba53d2de7fa37a4fb25bba7adbd69ad291f0e WatchSource:0}: Error finding container 8a58fb29c96750a7fea062e286cba53d2de7fa37a4fb25bba7adbd69ad291f0e: Status 404 returned error can't find the container with id 8a58fb29c96750a7fea062e286cba53d2de7fa37a4fb25bba7adbd69ad291f0e
Sep 17 19:03:08.759809 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:08.759773    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s" event={"ID":"5fdd73d8-efc1-4dae-a841-48ab4f4f521c","Type":"ContainerStarted","Data":"8d7647e65a69b711c090aa8dd240876e30319e488ad9f729896ed375fbf9ed35"}
Sep 17 19:03:08.759994 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:08.759836    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s" event={"ID":"5fdd73d8-efc1-4dae-a841-48ab4f4f521c","Type":"ContainerStarted","Data":"04453d6ec12b0e2168dac9f8a94ed75e6cf9c9e4b24c40498bb71975241db553"}
Sep 17 19:03:08.761122 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:08.761091    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd" event={"ID":"e999ce07-27aa-48c1-bddd-019e1c8fd731","Type":"ContainerStarted","Data":"8a58fb29c96750a7fea062e286cba53d2de7fa37a4fb25bba7adbd69ad291f0e"}
Sep 17 19:03:08.811181 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:08.811125    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s" podStartSLOduration=1.811106331 podStartE2EDuration="1.811106331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:03:08.809260048 +0000 UTC m=+3976.899881712" watchObservedRunningTime="2026-09-17 19:03:08.811106331 +0000 UTC m=+3976.901727981"
Sep 17 19:03:09.136930 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:09.136784    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s"
Sep 17 19:03:09.767338 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:09.767299    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd" event={"ID":"e999ce07-27aa-48c1-bddd-019e1c8fd731","Type":"ContainerStarted","Data":"1d471839933e4e1b7c406362a9987f1506ac2244870f7264324376697982246a"}
Sep 17 19:03:09.767558 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:09.767380    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd"
Sep 17 19:03:09.790431 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:09.790375    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd" podStartSLOduration=1.790358568 podStartE2EDuration="1.790358568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:03:09.788343865 +0000 UTC m=+3977.878965537" watchObservedRunningTime="2026-09-17 19:03:09.790358568 +0000 UTC m=+3977.880980217"
Sep 17 19:03:18.137119 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:18.137079    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s"
Sep 17 19:03:18.149807 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:18.149778    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s"
Sep 17 19:03:18.823033 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:18.823000    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s"
Sep 17 19:03:40.774948 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:40.774884    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd"
Sep 17 19:03:43.304935 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:43.304881    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd"]
Sep 17 19:03:43.344736 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:43.344701    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s"]
Sep 17 19:03:43.353716 ip-10-0-130-28 kubenswrapper[2581]: E0917 19:03:43.353661    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 19:03:43.353716 ip-10-0-130-28 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:03:43.353716 ip-10-0-130-28 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 19:03:43.353716 ip-10-0-130-28 kubenswrapper[2581]:  > containerID="8d7647e65a69b711c090aa8dd240876e30319e488ad9f729896ed375fbf9ed35" cmd=["/bin/sleep","15"]
Sep 17 19:03:43.354105 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:43.353735    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s" message="" err=<
Sep 17 19:03:43.354105 ip-10-0-130-28 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:03:43.354105 ip-10-0-130-28 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 19:03:43.354105 ip-10-0-130-28 kubenswrapper[2581]:  >
Sep 17 19:03:43.354105 ip-10-0-130-28 kubenswrapper[2581]: E0917 19:03:43.353748    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 19:03:43.354105 ip-10-0-130-28 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 19:03:43.354105 ip-10-0-130-28 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 17 19:03:43.354105 ip-10-0-130-28 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s" podUID="5fdd73d8-efc1-4dae-a841-48ab4f4f521c" containerName="main" containerID="cri-o://8d7647e65a69b711c090aa8dd240876e30319e488ad9f729896ed375fbf9ed35"
Sep 17 19:03:43.354105 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:43.353792    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s" podUID="5fdd73d8-efc1-4dae-a841-48ab4f4f521c" containerName="main" containerID="cri-o://8d7647e65a69b711c090aa8dd240876e30319e488ad9f729896ed375fbf9ed35" gracePeriod=60
Sep 17 19:03:43.715175 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:43.715145    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s"
Sep 17 19:03:43.859431 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:43.859331    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-model-cache\") pod \"5fdd73d8-efc1-4dae-a841-48ab4f4f521c\" (UID: \"5fdd73d8-efc1-4dae-a841-48ab4f4f521c\") "
Sep 17 19:03:43.859605 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:43.859435    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-tmp-dir\") pod \"5fdd73d8-efc1-4dae-a841-48ab4f4f521c\" (UID: \"5fdd73d8-efc1-4dae-a841-48ab4f4f521c\") "
Sep 17 19:03:43.859605 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:43.859485    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wj749\" (UniqueName: \"kubernetes.io/projected/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-kube-api-access-wj749\") pod \"5fdd73d8-efc1-4dae-a841-48ab4f4f521c\" (UID: \"5fdd73d8-efc1-4dae-a841-48ab4f4f521c\") "
Sep 17 19:03:43.859605 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:43.859506    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-dshm\") pod \"5fdd73d8-efc1-4dae-a841-48ab4f4f521c\" (UID: \"5fdd73d8-efc1-4dae-a841-48ab4f4f521c\") "
Sep 17 19:03:43.859605 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:43.859526    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-tls-certs\") pod \"5fdd73d8-efc1-4dae-a841-48ab4f4f521c\" (UID: \"5fdd73d8-efc1-4dae-a841-48ab4f4f521c\") "
Sep 17 19:03:43.859605 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:43.859546    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-home\") pod \"5fdd73d8-efc1-4dae-a841-48ab4f4f521c\" (UID: \"5fdd73d8-efc1-4dae-a841-48ab4f4f521c\") "
Sep 17 19:03:43.859896 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:43.859620    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-model-cache" (OuterVolumeSpecName: "model-cache") pod "5fdd73d8-efc1-4dae-a841-48ab4f4f521c" (UID: "5fdd73d8-efc1-4dae-a841-48ab4f4f521c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:03:43.859896 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:43.859727    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5fdd73d8-efc1-4dae-a841-48ab4f4f521c" (UID: "5fdd73d8-efc1-4dae-a841-48ab4f4f521c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:03:43.859896 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:43.859803    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-tmp-dir\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 19:03:43.859896 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:43.859823    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-model-cache\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 19:03:43.859896 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:43.859836    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-home" (OuterVolumeSpecName: "home") pod "5fdd73d8-efc1-4dae-a841-48ab4f4f521c" (UID: "5fdd73d8-efc1-4dae-a841-48ab4f4f521c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:03:43.861797 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:43.861765    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-dshm" (OuterVolumeSpecName: "dshm") pod "5fdd73d8-efc1-4dae-a841-48ab4f4f521c" (UID: "5fdd73d8-efc1-4dae-a841-48ab4f4f521c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:03:43.861917 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:43.861819    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-kube-api-access-wj749" (OuterVolumeSpecName: "kube-api-access-wj749") pod "5fdd73d8-efc1-4dae-a841-48ab4f4f521c" (UID: "5fdd73d8-efc1-4dae-a841-48ab4f4f521c"). InnerVolumeSpecName "kube-api-access-wj749". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:03:43.861917 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:43.861832    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5fdd73d8-efc1-4dae-a841-48ab4f4f521c" (UID: "5fdd73d8-efc1-4dae-a841-48ab4f4f521c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:03:43.928653 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:43.928616    2581 generic.go:356] "Generic (PLEG): container finished" podID="5fdd73d8-efc1-4dae-a841-48ab4f4f521c" containerID="8d7647e65a69b711c090aa8dd240876e30319e488ad9f729896ed375fbf9ed35" exitCode=0
Sep 17 19:03:43.928829 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:43.928664    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s"
Sep 17 19:03:43.928829 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:43.928702    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s" event={"ID":"5fdd73d8-efc1-4dae-a841-48ab4f4f521c","Type":"ContainerDied","Data":"8d7647e65a69b711c090aa8dd240876e30319e488ad9f729896ed375fbf9ed35"}
Sep 17 19:03:43.928829 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:43.928760    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s" event={"ID":"5fdd73d8-efc1-4dae-a841-48ab4f4f521c","Type":"ContainerDied","Data":"04453d6ec12b0e2168dac9f8a94ed75e6cf9c9e4b24c40498bb71975241db553"}
Sep 17 19:03:43.928829 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:43.928789    2581 scope.go:117] "RemoveContainer" containerID="8d7647e65a69b711c090aa8dd240876e30319e488ad9f729896ed375fbf9ed35"
Sep 17 19:03:43.937869 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:43.937847    2581 scope.go:117] "RemoveContainer" containerID="8d7647e65a69b711c090aa8dd240876e30319e488ad9f729896ed375fbf9ed35"
Sep 17 19:03:43.938181 ip-10-0-130-28 kubenswrapper[2581]: E0917 19:03:43.938161    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d7647e65a69b711c090aa8dd240876e30319e488ad9f729896ed375fbf9ed35\": container with ID starting with 8d7647e65a69b711c090aa8dd240876e30319e488ad9f729896ed375fbf9ed35 not found: ID does not exist" containerID="8d7647e65a69b711c090aa8dd240876e30319e488ad9f729896ed375fbf9ed35"
Sep 17 19:03:43.938246 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:43.938188    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d7647e65a69b711c090aa8dd240876e30319e488ad9f729896ed375fbf9ed35"} err="failed to get container status \"8d7647e65a69b711c090aa8dd240876e30319e488ad9f729896ed375fbf9ed35\": rpc error: code = NotFound desc = could not find container \"8d7647e65a69b711c090aa8dd240876e30319e488ad9f729896ed375fbf9ed35\": container with ID starting with 8d7647e65a69b711c090aa8dd240876e30319e488ad9f729896ed375fbf9ed35 not found: ID does not exist"
Sep 17 19:03:43.960694 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:43.960652    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-home\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 19:03:43.960694 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:43.960688    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj749\" (UniqueName: \"kubernetes.io/projected/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-kube-api-access-wj749\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 19:03:43.960871 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:43.960704    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-dshm\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 19:03:43.960871 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:43.960716    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fdd73d8-efc1-4dae-a841-48ab4f4f521c-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 19:03:43.987989 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:43.987949    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s"]
Sep 17 19:03:43.993783 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:43.993758    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-4kf4s"]
Sep 17 19:03:44.358882 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:44.358846    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fdd73d8-efc1-4dae-a841-48ab4f4f521c" path="/var/lib/kubelet/pods/5fdd73d8-efc1-4dae-a841-48ab4f4f521c/volumes"
Sep 17 19:03:50.775172 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:50.775131    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd" podUID="e999ce07-27aa-48c1-bddd-019e1c8fd731" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:03:58.324293 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:03:58.324220    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd" podUID="e999ce07-27aa-48c1-bddd-019e1c8fd731" containerName="main" containerID="cri-o://1d471839933e4e1b7c406362a9987f1506ac2244870f7264324376697982246a" gracePeriod=45
Sep 17 19:04:00.775107 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:04:00.775061    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd" podUID="e999ce07-27aa-48c1-bddd-019e1c8fd731" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:04:10.775011 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:04:10.774969    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd" podUID="e999ce07-27aa-48c1-bddd-019e1c8fd731" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:04:10.775428 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:04:10.775074    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd"
Sep 17 19:04:20.774431 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:04:20.774342    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd" podUID="e999ce07-27aa-48c1-bddd-019e1c8fd731" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 19:04:28.572324 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:04:28.572294    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd"
Sep 17 19:04:28.658215 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:04:28.658119    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4n7df\" (UniqueName: \"kubernetes.io/projected/e999ce07-27aa-48c1-bddd-019e1c8fd731-kube-api-access-4n7df\") pod \"e999ce07-27aa-48c1-bddd-019e1c8fd731\" (UID: \"e999ce07-27aa-48c1-bddd-019e1c8fd731\") "
Sep 17 19:04:28.658371 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:04:28.658249    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e999ce07-27aa-48c1-bddd-019e1c8fd731-tls-certs\") pod \"e999ce07-27aa-48c1-bddd-019e1c8fd731\" (UID: \"e999ce07-27aa-48c1-bddd-019e1c8fd731\") "
Sep 17 19:04:28.660377 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:04:28.660343    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e999ce07-27aa-48c1-bddd-019e1c8fd731-kube-api-access-4n7df" (OuterVolumeSpecName: "kube-api-access-4n7df") pod "e999ce07-27aa-48c1-bddd-019e1c8fd731" (UID: "e999ce07-27aa-48c1-bddd-019e1c8fd731"). InnerVolumeSpecName "kube-api-access-4n7df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:04:28.660506 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:04:28.660447    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e999ce07-27aa-48c1-bddd-019e1c8fd731-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e999ce07-27aa-48c1-bddd-019e1c8fd731" (UID: "e999ce07-27aa-48c1-bddd-019e1c8fd731"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:04:28.759422 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:04:28.759374    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n7df\" (UniqueName: \"kubernetes.io/projected/e999ce07-27aa-48c1-bddd-019e1c8fd731-kube-api-access-4n7df\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 19:04:28.759422 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:04:28.759416    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e999ce07-27aa-48c1-bddd-019e1c8fd731-tls-certs\") on node \"ip-10-0-130-28.ec2.internal\" DevicePath \"\""
Sep 17 19:04:29.143087 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:04:29.143049    2581 generic.go:356] "Generic (PLEG): container finished" podID="e999ce07-27aa-48c1-bddd-019e1c8fd731" containerID="1d471839933e4e1b7c406362a9987f1506ac2244870f7264324376697982246a" exitCode=0
Sep 17 19:04:29.143282 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:04:29.143098    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd"
Sep 17 19:04:29.143282 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:04:29.143141    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd" event={"ID":"e999ce07-27aa-48c1-bddd-019e1c8fd731","Type":"ContainerDied","Data":"1d471839933e4e1b7c406362a9987f1506ac2244870f7264324376697982246a"}
Sep 17 19:04:29.143282 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:04:29.143204    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd" event={"ID":"e999ce07-27aa-48c1-bddd-019e1c8fd731","Type":"ContainerDied","Data":"8a58fb29c96750a7fea062e286cba53d2de7fa37a4fb25bba7adbd69ad291f0e"}
Sep 17 19:04:29.143282 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:04:29.143235    2581 scope.go:117] "RemoveContainer" containerID="1d471839933e4e1b7c406362a9987f1506ac2244870f7264324376697982246a"
Sep 17 19:04:29.152508 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:04:29.152488    2581 scope.go:117] "RemoveContainer" containerID="1d471839933e4e1b7c406362a9987f1506ac2244870f7264324376697982246a"
Sep 17 19:04:29.152779 ip-10-0-130-28 kubenswrapper[2581]: E0917 19:04:29.152755    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d471839933e4e1b7c406362a9987f1506ac2244870f7264324376697982246a\": container with ID starting with 1d471839933e4e1b7c406362a9987f1506ac2244870f7264324376697982246a not found: ID does not exist" containerID="1d471839933e4e1b7c406362a9987f1506ac2244870f7264324376697982246a"
Sep 17 19:04:29.152870 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:04:29.152788    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d471839933e4e1b7c406362a9987f1506ac2244870f7264324376697982246a"} err="failed to get container status \"1d471839933e4e1b7c406362a9987f1506ac2244870f7264324376697982246a\": rpc error: code = NotFound desc = could not find container \"1d471839933e4e1b7c406362a9987f1506ac2244870f7264324376697982246a\": container with ID starting with 1d471839933e4e1b7c406362a9987f1506ac2244870f7264324376697982246a not found: ID does not exist"
Sep 17 19:04:29.166967 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:04:29.166941    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd"]
Sep 17 19:04:29.172417 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:04:29.172391    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s2nqd"]
Sep 17 19:04:30.357283 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:04:30.357244    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e999ce07-27aa-48c1-bddd-019e1c8fd731" path="/var/lib/kubelet/pods/e999ce07-27aa-48c1-bddd-019e1c8fd731/volumes"
Sep 17 19:05:01.117953 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:01.117913    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw_8421b2a7-bdb3-4dbd-af93-943395f1007f/istio-proxy/0.log"
Sep 17 19:05:01.122330 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:01.122305    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw_8421b2a7-bdb3-4dbd-af93-943395f1007f/istio-proxy/1.log"
Sep 17 19:05:02.224572 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:02.224528    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw_8421b2a7-bdb3-4dbd-af93-943395f1007f/istio-proxy/0.log"
Sep 17 19:05:02.229043 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:02.229016    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw_8421b2a7-bdb3-4dbd-af93-943395f1007f/istio-proxy/1.log"
Sep 17 19:05:03.334101 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:03.334064    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-dv575_d33c29fb-9b85-4c2c-a933-a75967180a8d/authorino/0.log"
Sep 17 19:05:03.586954 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:03.586860    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-d5ft4_e64ea2e8-c9d8-4c78-8649-ebd887d79399/kuadrant-console-plugin/0.log"
Sep 17 19:05:08.177129 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:08.177086    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f4j9s/must-gather-fvg2j"]
Sep 17 19:05:08.177638 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:08.177616    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fdd73d8-efc1-4dae-a841-48ab4f4f521c" containerName="main"
Sep 17 19:05:08.177720 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:08.177642    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fdd73d8-efc1-4dae-a841-48ab4f4f521c" containerName="main"
Sep 17 19:05:08.177720 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:08.177666    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e999ce07-27aa-48c1-bddd-019e1c8fd731" containerName="main"
Sep 17 19:05:08.177720 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:08.177674    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e999ce07-27aa-48c1-bddd-019e1c8fd731" containerName="main"
Sep 17 19:05:08.177876 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:08.177788    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fdd73d8-efc1-4dae-a841-48ab4f4f521c" containerName="main"
Sep 17 19:05:08.177876 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:08.177801    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e999ce07-27aa-48c1-bddd-019e1c8fd731" containerName="main"
Sep 17 19:05:08.181974 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:08.181951    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f4j9s/must-gather-fvg2j"
Sep 17 19:05:08.187345 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:08.187304    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-f4j9s\"/\"default-dockercfg-9g8bv\""
Sep 17 19:05:08.187498 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:08.187399    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f4j9s\"/\"openshift-service-ca.crt\""
Sep 17 19:05:08.187596 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:08.187582    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f4j9s\"/\"kube-root-ca.crt\""
Sep 17 19:05:08.202220 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:08.202188    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f4j9s/must-gather-fvg2j"]
Sep 17 19:05:08.291763 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:08.291725    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/791d5252-138d-4237-aee7-ed00a2de0626-must-gather-output\") pod \"must-gather-fvg2j\" (UID: \"791d5252-138d-4237-aee7-ed00a2de0626\") " pod="openshift-must-gather-f4j9s/must-gather-fvg2j"
Sep 17 19:05:08.291989 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:08.291780    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sw5k\" (UniqueName: \"kubernetes.io/projected/791d5252-138d-4237-aee7-ed00a2de0626-kube-api-access-4sw5k\") pod \"must-gather-fvg2j\" (UID: \"791d5252-138d-4237-aee7-ed00a2de0626\") " pod="openshift-must-gather-f4j9s/must-gather-fvg2j"
Sep 17 19:05:08.392632 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:08.392595    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/791d5252-138d-4237-aee7-ed00a2de0626-must-gather-output\") pod \"must-gather-fvg2j\" (UID: \"791d5252-138d-4237-aee7-ed00a2de0626\") " pod="openshift-must-gather-f4j9s/must-gather-fvg2j"
Sep 17 19:05:08.392789 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:08.392650    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sw5k\" (UniqueName: \"kubernetes.io/projected/791d5252-138d-4237-aee7-ed00a2de0626-kube-api-access-4sw5k\") pod \"must-gather-fvg2j\" (UID: \"791d5252-138d-4237-aee7-ed00a2de0626\") " pod="openshift-must-gather-f4j9s/must-gather-fvg2j"
Sep 17 19:05:08.393045 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:08.393022    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/791d5252-138d-4237-aee7-ed00a2de0626-must-gather-output\") pod \"must-gather-fvg2j\" (UID: \"791d5252-138d-4237-aee7-ed00a2de0626\") " pod="openshift-must-gather-f4j9s/must-gather-fvg2j"
Sep 17 19:05:08.432781 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:08.432709    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sw5k\" (UniqueName: \"kubernetes.io/projected/791d5252-138d-4237-aee7-ed00a2de0626-kube-api-access-4sw5k\") pod \"must-gather-fvg2j\" (UID: \"791d5252-138d-4237-aee7-ed00a2de0626\") " pod="openshift-must-gather-f4j9s/must-gather-fvg2j"
Sep 17 19:05:08.491461 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:08.491427    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f4j9s/must-gather-fvg2j"
Sep 17 19:05:08.785736 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:08.785695    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f4j9s/must-gather-fvg2j"]
Sep 17 19:05:08.790515 ip-10-0-130-28 kubenswrapper[2581]: W0917 19:05:08.790479    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod791d5252_138d_4237_aee7_ed00a2de0626.slice/crio-72c0bf501e3dcd4fe7d3aeeed1f99440f291b4ea3cc1d5e2798abf6dd38fd693 WatchSource:0}: Error finding container 72c0bf501e3dcd4fe7d3aeeed1f99440f291b4ea3cc1d5e2798abf6dd38fd693: Status 404 returned error can't find the container with id 72c0bf501e3dcd4fe7d3aeeed1f99440f291b4ea3cc1d5e2798abf6dd38fd693
Sep 17 19:05:08.792727 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:08.792704    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:05:09.327098 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:09.327060    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f4j9s/must-gather-fvg2j" event={"ID":"791d5252-138d-4237-aee7-ed00a2de0626","Type":"ContainerStarted","Data":"72c0bf501e3dcd4fe7d3aeeed1f99440f291b4ea3cc1d5e2798abf6dd38fd693"}
Sep 17 19:05:10.334779 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:10.334738    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f4j9s/must-gather-fvg2j" event={"ID":"791d5252-138d-4237-aee7-ed00a2de0626","Type":"ContainerStarted","Data":"2e458794d0345e10bc45209f982d46aac21693f6238526b323cc119c1871c4a4"}
Sep 17 19:05:10.335194 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:10.334800    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f4j9s/must-gather-fvg2j" event={"ID":"791d5252-138d-4237-aee7-ed00a2de0626","Type":"ContainerStarted","Data":"0734ce546e2995c00e6e6593f325561bcd49e37b5f14ee95f739c3372a8ef7c5"}
Sep 17 19:05:10.362639 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:10.362585    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-f4j9s/must-gather-fvg2j" podStartSLOduration=1.403506464 podStartE2EDuration="2.362568096s" totalImagesPullingTime="959.061632ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:05:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:05:10.362399006 +0000 UTC m=+4098.453020657" watchObservedRunningTime="2026-09-17 19:05:10.362568096 +0000 UTC m=+4098.453189746"
Sep 17 19:05:11.999155 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:11.999120    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9xft5_33a2724c-596d-416e-99e2-583a9db84dcf/global-pull-secret-syncer/0.log"
Sep 17 19:05:12.413662 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:12.413526    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8tpnw_0f73bdd3-5b33-4d88-9d8b-eb540633a58e/konnectivity-agent/0.log"
Sep 17 19:05:12.685751 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:12.685664    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-28.ec2.internal_672f4fffca246081b3733694dc0095b6/haproxy/0.log"
Sep 17 19:05:16.585922 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:16.585768    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-dv575_d33c29fb-9b85-4c2c-a933-a75967180a8d/authorino/0.log"
Sep 17 19:05:17.250435 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:17.250397    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-d5ft4_e64ea2e8-c9d8-4c78-8649-ebd887d79399/kuadrant-console-plugin/0.log"
Sep 17 19:05:20.336247 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:20.336188    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_61ef0b04-cf59-4219-99d1-a30a40477762/alertmanager/0.log"
Sep 17 19:05:20.420140 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:20.420110    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_61ef0b04-cf59-4219-99d1-a30a40477762/config-reloader/0.log"
Sep 17 19:05:20.502974 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:20.502939    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_61ef0b04-cf59-4219-99d1-a30a40477762/kube-rbac-proxy-web/0.log"
Sep 17 19:05:20.587040 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:20.586942    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_61ef0b04-cf59-4219-99d1-a30a40477762/kube-rbac-proxy/0.log"
Sep 17 19:05:20.682551 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:20.682517    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_61ef0b04-cf59-4219-99d1-a30a40477762/kube-rbac-proxy-metric/0.log"
Sep 17 19:05:20.778520 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:20.778489    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_61ef0b04-cf59-4219-99d1-a30a40477762/prom-label-proxy/0.log"
Sep 17 19:05:20.857316 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:20.857219    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_61ef0b04-cf59-4219-99d1-a30a40477762/init-config-reloader/0.log"
Sep 17 19:05:20.962504 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:20.962466    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-zt5hc_9a99ade3-9dfb-4718-bf9b-07d713715bab/cluster-monitoring-operator/0.log"
Sep 17 19:05:21.066424 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:21.066385    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-7tlp8_361d0f4d-9529-461d-b89e-8e0cd8ca67ec/kube-state-metrics/0.log"
Sep 17 19:05:21.182207 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:21.182094    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-7tlp8_361d0f4d-9529-461d-b89e-8e0cd8ca67ec/kube-rbac-proxy-main/0.log"
Sep 17 19:05:21.257060 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:21.257009    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-7tlp8_361d0f4d-9529-461d-b89e-8e0cd8ca67ec/kube-rbac-proxy-self/0.log"
Sep 17 19:05:21.380095 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:21.380054    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f4j9s/perf-node-gather-daemonset-htjb4"]
Sep 17 19:05:21.384310 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:21.384285    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-htjb4"
Sep 17 19:05:21.407008 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:21.406973    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f4j9s/perf-node-gather-daemonset-htjb4"]
Sep 17 19:05:21.421403 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:21.421369    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6bd7bb56fc-7q69l_6c7b7223-42cd-4b34-ab15-7f9953d22478/metrics-server/0.log"
Sep 17 19:05:21.524575 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:21.524527    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ba5f89d-91c2-4e1d-ae96-2534dfedd5c9-sys\") pod \"perf-node-gather-daemonset-htjb4\" (UID: \"8ba5f89d-91c2-4e1d-ae96-2534dfedd5c9\") " pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-htjb4"
Sep 17 19:05:21.524768 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:21.524596    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt5c5\" (UniqueName: \"kubernetes.io/projected/8ba5f89d-91c2-4e1d-ae96-2534dfedd5c9-kube-api-access-vt5c5\") pod \"perf-node-gather-daemonset-htjb4\" (UID: \"8ba5f89d-91c2-4e1d-ae96-2534dfedd5c9\") " pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-htjb4"
Sep 17 19:05:21.524768 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:21.524631    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8ba5f89d-91c2-4e1d-ae96-2534dfedd5c9-proc\") pod \"perf-node-gather-daemonset-htjb4\" (UID: \"8ba5f89d-91c2-4e1d-ae96-2534dfedd5c9\") " pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-htjb4"
Sep 17 19:05:21.524768 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:21.524650    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ba5f89d-91c2-4e1d-ae96-2534dfedd5c9-lib-modules\") pod \"perf-node-gather-daemonset-htjb4\" (UID: \"8ba5f89d-91c2-4e1d-ae96-2534dfedd5c9\") " pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-htjb4"
Sep 17 19:05:21.524768 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:21.524712    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8ba5f89d-91c2-4e1d-ae96-2534dfedd5c9-podres\") pod \"perf-node-gather-daemonset-htjb4\" (UID: \"8ba5f89d-91c2-4e1d-ae96-2534dfedd5c9\") " pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-htjb4"
Sep 17 19:05:21.617950 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:21.617892    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5fr6x_442e5462-5952-46bc-819b-2aba220bea98/node-exporter/0.log"
Sep 17 19:05:21.625978 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:21.625944    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt5c5\" (UniqueName: \"kubernetes.io/projected/8ba5f89d-91c2-4e1d-ae96-2534dfedd5c9-kube-api-access-vt5c5\") pod \"perf-node-gather-daemonset-htjb4\" (UID: \"8ba5f89d-91c2-4e1d-ae96-2534dfedd5c9\") " pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-htjb4"
Sep 17 19:05:21.626141 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:21.626013    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8ba5f89d-91c2-4e1d-ae96-2534dfedd5c9-proc\") pod \"perf-node-gather-daemonset-htjb4\" (UID: \"8ba5f89d-91c2-4e1d-ae96-2534dfedd5c9\") " pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-htjb4"
Sep 17 19:05:21.626141 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:21.626048    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ba5f89d-91c2-4e1d-ae96-2534dfedd5c9-lib-modules\") pod \"perf-node-gather-daemonset-htjb4\" (UID: \"8ba5f89d-91c2-4e1d-ae96-2534dfedd5c9\") " pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-htjb4"
Sep 17 19:05:21.626141 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:21.626109    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8ba5f89d-91c2-4e1d-ae96-2534dfedd5c9-podres\") pod \"perf-node-gather-daemonset-htjb4\" (UID: \"8ba5f89d-91c2-4e1d-ae96-2534dfedd5c9\") " pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-htjb4"
Sep 17 19:05:21.626301 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:21.626142    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8ba5f89d-91c2-4e1d-ae96-2534dfedd5c9-proc\") pod \"perf-node-gather-daemonset-htjb4\" (UID: \"8ba5f89d-91c2-4e1d-ae96-2534dfedd5c9\") " pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-htjb4"
Sep 17 19:05:21.626301 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:21.626171    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ba5f89d-91c2-4e1d-ae96-2534dfedd5c9-sys\") pod \"perf-node-gather-daemonset-htjb4\" (UID: \"8ba5f89d-91c2-4e1d-ae96-2534dfedd5c9\") " pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-htjb4"
Sep 17 19:05:21.626301 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:21.626255    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ba5f89d-91c2-4e1d-ae96-2534dfedd5c9-sys\") pod \"perf-node-gather-daemonset-htjb4\" (UID: \"8ba5f89d-91c2-4e1d-ae96-2534dfedd5c9\") " pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-htjb4"
Sep 17 19:05:21.626301 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:21.626269    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8ba5f89d-91c2-4e1d-ae96-2534dfedd5c9-podres\") pod \"perf-node-gather-daemonset-htjb4\" (UID: \"8ba5f89d-91c2-4e1d-ae96-2534dfedd5c9\") " pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-htjb4"
Sep 17 19:05:21.626301 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:21.626272    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ba5f89d-91c2-4e1d-ae96-2534dfedd5c9-lib-modules\") pod \"perf-node-gather-daemonset-htjb4\" (UID: \"8ba5f89d-91c2-4e1d-ae96-2534dfedd5c9\") " pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-htjb4"
Sep 17 19:05:21.642060 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:21.640247    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt5c5\" (UniqueName: \"kubernetes.io/projected/8ba5f89d-91c2-4e1d-ae96-2534dfedd5c9-kube-api-access-vt5c5\") pod \"perf-node-gather-daemonset-htjb4\" (UID: \"8ba5f89d-91c2-4e1d-ae96-2534dfedd5c9\") " pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-htjb4"
Sep 17 19:05:21.687034 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:21.687000    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5fr6x_442e5462-5952-46bc-819b-2aba220bea98/kube-rbac-proxy/0.log"
Sep 17 19:05:21.695696 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:21.695662    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-htjb4"
Sep 17 19:05:21.757674 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:21.757506    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5fr6x_442e5462-5952-46bc-819b-2aba220bea98/init-textfile/0.log"
Sep 17 19:05:22.003166 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:22.002946    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f4j9s/perf-node-gather-daemonset-htjb4"]
Sep 17 19:05:22.005383 ip-10-0-130-28 kubenswrapper[2581]: W0917 19:05:22.005352    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8ba5f89d_91c2_4e1d_ae96_2534dfedd5c9.slice/crio-df0fbe3b0a79c8cdcffbe16d51cabee92bdbea57644ee13780c0ca9ce5643366 WatchSource:0}: Error finding container df0fbe3b0a79c8cdcffbe16d51cabee92bdbea57644ee13780c0ca9ce5643366: Status 404 returned error can't find the container with id df0fbe3b0a79c8cdcffbe16d51cabee92bdbea57644ee13780c0ca9ce5643366
Sep 17 19:05:22.340761 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:22.340649    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-2gv9q_f8069ed7-c6a1-41cf-ba38-f21a714557e7/kube-rbac-proxy-main/0.log"
Sep 17 19:05:22.401690 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:22.401614    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-2gv9q_f8069ed7-c6a1-41cf-ba38-f21a714557e7/kube-rbac-proxy-self/0.log"
Sep 17 19:05:22.425753 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:22.425715    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-htjb4" event={"ID":"8ba5f89d-91c2-4e1d-ae96-2534dfedd5c9","Type":"ContainerStarted","Data":"d79f1bf903cb8e5e46fc4106a3f5d8af2ab382c1f4cdcdd4584c8291cd2d3269"}
Sep 17 19:05:22.425988 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:22.425778    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-htjb4" event={"ID":"8ba5f89d-91c2-4e1d-ae96-2534dfedd5c9","Type":"ContainerStarted","Data":"df0fbe3b0a79c8cdcffbe16d51cabee92bdbea57644ee13780c0ca9ce5643366"}
Sep 17 19:05:22.425988 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:22.425831    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-htjb4"
Sep 17 19:05:22.471769 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:22.471706    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-htjb4" podStartSLOduration=1.471688139 podStartE2EDuration="1.471688139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:05:22.470321606 +0000 UTC m=+4110.560943258" watchObservedRunningTime="2026-09-17 19:05:22.471688139 +0000 UTC m=+4110.562309790"
Sep 17 19:05:22.492953 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:22.492914    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-2gv9q_f8069ed7-c6a1-41cf-ba38-f21a714557e7/openshift-state-metrics/0.log"
Sep 17 19:05:23.302048 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:23.302012    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-598c547cf5-x5628_ca913e5c-28e7-4ef8-9e6a-34c90557c838/telemeter-client/0.log"
Sep 17 19:05:23.352156 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:23.352123    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-598c547cf5-x5628_ca913e5c-28e7-4ef8-9e6a-34c90557c838/reload/0.log"
Sep 17 19:05:23.407282 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:23.407247    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-598c547cf5-x5628_ca913e5c-28e7-4ef8-9e6a-34c90557c838/kube-rbac-proxy/0.log"
Sep 17 19:05:23.465938 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:23.465885    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d69786d4-cqbxm_ab31f5c3-3c86-4de6-b021-e56291f4ce44/thanos-query/0.log"
Sep 17 19:05:23.540467 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:23.540414    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d69786d4-cqbxm_ab31f5c3-3c86-4de6-b021-e56291f4ce44/kube-rbac-proxy-web/0.log"
Sep 17 19:05:23.591652 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:23.591557    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d69786d4-cqbxm_ab31f5c3-3c86-4de6-b021-e56291f4ce44/kube-rbac-proxy/0.log"
Sep 17 19:05:23.644749 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:23.644711    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d69786d4-cqbxm_ab31f5c3-3c86-4de6-b021-e56291f4ce44/prom-label-proxy/0.log"
Sep 17 19:05:23.704770 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:23.704737    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d69786d4-cqbxm_ab31f5c3-3c86-4de6-b021-e56291f4ce44/kube-rbac-proxy-rules/0.log"
Sep 17 19:05:23.805842 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:23.805803    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d69786d4-cqbxm_ab31f5c3-3c86-4de6-b021-e56291f4ce44/kube-rbac-proxy-metrics/0.log"
Sep 17 19:05:25.702324 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:25.702289    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/2.log"
Sep 17 19:05:25.710562 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:25.710523    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-l5vq7_b41e1ab1-9be9-4db0-82da-f3dc6d863a24/console-operator/3.log"
Sep 17 19:05:28.441743 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:28.441713    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-f4j9s/perf-node-gather-daemonset-htjb4"
Sep 17 19:05:29.353726 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:29.353695    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sgwxz_13c59508-5d4e-4ea8-a211-42ec8c5bfffa/dns/0.log"
Sep 17 19:05:29.396862 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:29.396828    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sgwxz_13c59508-5d4e-4ea8-a211-42ec8c5bfffa/kube-rbac-proxy/0.log"
Sep 17 19:05:29.524717 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:29.524684    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qcqn9_e73ac43a-8a55-40ea-902c-5b06dfcc5d0b/dns-node-resolver/0.log"
Sep 17 19:05:30.711838 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:30.711788    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zsz9h_4383fe04-b7e0-4dd9-b3cc-ddca3d57aed2/node-ca/0.log"
Sep 17 19:05:32.384687 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:32.384645    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw_8421b2a7-bdb3-4dbd-af93-943395f1007f/istio-proxy/0.log"
Sep 17 19:05:32.389492 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:32.389461    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw_8421b2a7-bdb3-4dbd-af93-943395f1007f/istio-proxy/1.log"
Sep 17 19:05:32.731427 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:32.731385    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw_8421b2a7-bdb3-4dbd-af93-943395f1007f/istio-proxy/0.log"
Sep 17 19:05:32.736279 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:32.736248    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw_8421b2a7-bdb3-4dbd-af93-943395f1007f/istio-proxy/1.log"
Sep 17 19:05:33.749440 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:33.749401    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6nv7n_06dbe6ba-8122-4c23-9dbe-63e2de449f7c/serve-healthcheck-canary/0.log"
Sep 17 19:05:33.832490 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:33.832454    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw_8421b2a7-bdb3-4dbd-af93-943395f1007f/istio-proxy/0.log"
Sep 17 19:05:33.838006 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:33.837974    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-58c5cd85c9-2xpqw_8421b2a7-bdb3-4dbd-af93-943395f1007f/istio-proxy/1.log"
Sep 17 19:05:35.526376 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:35.526345    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rw8pw_13b39edd-14cc-42ac-b8e6-e6c61ae9018e/kube-rbac-proxy/0.log"
Sep 17 19:05:35.581844 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:35.581809    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rw8pw_13b39edd-14cc-42ac-b8e6-e6c61ae9018e/exporter/0.log"
Sep 17 19:05:35.631531 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:35.631496    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rw8pw_13b39edd-14cc-42ac-b8e6-e6c61ae9018e/extractor/0.log"
Sep 17 19:05:38.891946 ip-10-0-130-28 kubenswrapper[2581]: I0917 19:05:38.891883    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5654b77c54-7n5n5_71d2bec5-4ce3-4c5a-bc52-78cc8fb08f6c/manager/0.log"