Sep 17 17:32:32.800139 ip-10-0-141-30 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 17:32:32.800149 ip-10-0-141-30 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 17:32:32.800157 ip-10-0-141-30 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 17:32:32.800310 ip-10-0-141-30 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 17:32:43.014124 ip-10-0-141-30 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 17:32:43.014139 ip-10-0-141-30 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot c9de11b9052e4fe19ea1ae5886cb3851 --
Sep 17 17:34:47.439936 ip-10-0-141-30 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 17:34:50.820001 ip-10-0-141-30 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:34:50.820001 ip-10-0-141-30 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 17:34:50.820001 ip-10-0-141-30 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:34:50.820001 ip-10-0-141-30 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 17:34:50.820001 ip-10-0-141-30 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:34:50.820947 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.820082    2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 17:34:50.822880 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822865    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:50.822880 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822879    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:50.822940 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822882    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:50.822940 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822887    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:50.822940 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822891    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:50.822940 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822895    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:50.822940 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822898    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:50.822940 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822901    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:50.822940 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822904    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:50.822940 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822908    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:50.822940 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822910    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:50.822940 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822913    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:50.822940 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822916    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:50.822940 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822918    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:50.822940 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822921    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:50.822940 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822928    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:50.822940 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822931    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:50.822940 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822935    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:50.822940 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822937    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:50.822940 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822940    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:50.822940 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822943    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:50.823405 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822947    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:50.823405 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822952    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:50.823405 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822955    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:50.823405 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822958    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:50.823405 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822961    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:50.823405 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822964    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:50.823405 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822967    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:50.823405 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822970    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:50.823405 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822973    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:50.823405 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822975    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:50.823405 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822978    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:50.823405 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822981    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:50.823405 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822983    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:50.823405 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822986    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:50.823405 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822988    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:50.823405 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822991    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:50.823405 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822994    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:50.823405 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822996    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:50.823405 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.822999    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:50.823887 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823001    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:50.823887 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823004    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:50.823887 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823006    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:50.823887 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823009    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:50.823887 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823011    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:50.823887 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823014    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:50.823887 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823018    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:50.823887 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823021    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:50.823887 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823024    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:50.823887 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823027    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:50.823887 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823029    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:50.823887 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823032    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:50.823887 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823034    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:50.823887 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823037    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:50.823887 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823040    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:50.823887 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823043    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:50.823887 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823046    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:50.823887 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823048    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:50.823887 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823051    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:50.824345 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823054    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:50.824345 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823056    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:50.824345 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823059    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:50.824345 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823061    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:50.824345 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823064    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:50.824345 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823067    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:50.824345 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823069    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:50.824345 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823073    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:50.824345 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823076    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:50.824345 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823078    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:50.824345 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823080    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:50.824345 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823083    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:50.824345 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823086    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:50.824345 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823088    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:50.824345 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823090    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:50.824345 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823093    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:50.824345 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823096    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:50.824345 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823099    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:50.824345 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823101    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:50.824345 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823104    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:50.824843 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823106    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:50.824843 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823109    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:50.824843 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823111    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:50.824843 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823114    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:50.824843 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823116    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:50.824843 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823119    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:50.824843 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823121    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:50.824843 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823124    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:50.824843 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823127    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:50.824843 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823130    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:50.824843 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823132    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:50.824843 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823135    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:50.824843 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823137    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:50.824843 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823140    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:50.824843 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823143    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:50.824843 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823145    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:50.824843 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823148    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:50.824843 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823150    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:50.824843 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823153    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:50.824843 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823156    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:50.824843 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823855    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:50.825346 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823862    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:50.825346 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823865    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:50.825346 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823867    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:50.825346 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823870    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:50.825346 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823873    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:50.825346 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823876    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:50.825346 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823879    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:50.825346 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823881    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:50.825346 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823885    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:50.825346 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823887    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:50.825346 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823891    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:50.825346 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823895    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:50.825346 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823898    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:50.825346 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823901    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:50.825346 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823904    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:50.825346 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823906    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:50.825346 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823909    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:50.825346 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823912    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:50.825346 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823916    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:50.825346 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823919    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:50.825860 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823922    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:50.825860 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823925    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:50.825860 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823928    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:50.825860 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823931    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:50.825860 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823934    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:50.825860 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823936    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:50.825860 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823939    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:50.825860 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823942    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:50.825860 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823944    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:50.825860 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823947    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:50.825860 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823949    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:50.825860 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823952    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:50.825860 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823954    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:50.825860 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823957    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:50.825860 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823959    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:50.825860 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823962    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:50.825860 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823964    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:50.825860 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823967    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:50.825860 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823970    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:50.825860 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823973    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:50.826354 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823977    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:50.826354 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823981    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:50.826354 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823984    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:50.826354 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823987    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:50.826354 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823990    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:50.826354 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823993    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:50.826354 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823996    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:50.826354 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.823999    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:50.826354 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824002    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:50.826354 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824004    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:50.826354 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824008    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:50.826354 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824010    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:50.826354 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824013    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:50.826354 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824016    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:50.826354 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824020    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:50.826354 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824023    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:50.826354 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824027    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:50.826354 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824030    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:50.826354 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824033    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:50.826892 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824035    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:50.826892 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824038    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:50.826892 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824040    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:50.826892 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824043    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:50.826892 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824045    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:50.826892 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824048    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:50.826892 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824050    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:50.826892 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824053    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:50.826892 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824055    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:50.826892 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824058    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:50.826892 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824061    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:50.826892 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824063    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:50.826892 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824066    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:50.826892 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824068    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:50.826892 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824071    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:50.826892 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824074    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:50.826892 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824077    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:50.826892 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824079    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:50.826892 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824082    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:50.827369 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824085    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:50.827369 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824087    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:50.827369 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824090    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:50.827369 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824092    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:50.827369 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824095    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:50.827369 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824098    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:50.827369 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824101    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:50.827369 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824103    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:50.827369 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824106    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:50.827369 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824109    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:50.827369 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824111    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:50.827369 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824113    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:50.827369 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824116    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:50.827369 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824118    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:50.827369 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824121    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:50.827369 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824124    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:50.827369 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824126    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:50.827369 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824129    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:50.827369 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824131    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:50.827369 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824134    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:50.827874 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824522    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:50.827874 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824527    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:50.827874 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824531    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:50.827874 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824534    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:50.827874 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824537    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:50.827874 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824540    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:50.827874 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824542    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:50.827874 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824545    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:50.827874 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824548    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:50.827874 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824551    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:50.827874 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824554    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:50.827874 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824556    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:50.827874 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824559    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:50.827874 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824561    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:50.827874 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824565    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:50.827874 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824567    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:50.827874 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824570    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:50.827874 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824573    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:50.827874 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824576    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:50.827874 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824578    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:50.828378 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824581    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:50.828378 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824584    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:50.828378 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824587    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:50.828378 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824591    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:50.828378 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824593    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:50.828378 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824596    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:50.828378 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824599    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:50.828378 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824603    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:50.828378 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824606    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:50.828378 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824609    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:50.828378 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824611    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:50.828378 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824614    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:50.828378 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824617    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:50.828378 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824619    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:50.828378 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824622    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:50.828378 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824624    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:50.828378 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824627    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:50.828378 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824629    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:50.828378 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824632    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:50.828866 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824634    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:50.828866 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824637    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:50.828866 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824640    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:50.828866 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824642    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:50.828866 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824645    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:50.828866 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824648    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:50.828866 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824651    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:50.828866 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824653    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:50.828866 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824672    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:50.828866 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824675    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:50.828866 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824678    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:50.828866 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824680    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:50.828866 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824683    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:50.828866 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824685    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:50.828866 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824688    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:50.828866 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824690    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:50.828866 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824693    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:50.828866 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824696    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:50.828866 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824698    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:50.828866 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824701    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:50.829380 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824703    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:50.829380 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824706    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:50.829380 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824709    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:50.829380 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824711    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:50.829380 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824714    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:50.829380 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824717    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:50.829380 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824720    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:50.829380 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824722    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:50.829380 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824727    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:50.829380 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824730    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:50.829380 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824733    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:50.829380 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824736    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:50.829380 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824739    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:50.829380 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824742    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:50.829380 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824745    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:50.829380 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824748    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:50.829380 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824751    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:50.829380 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824753    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:50.829380 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824758    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:50.829852 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824761    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:50.829852 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824764    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:50.829852 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824767    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:50.829852 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824770    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:50.829852 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824773    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:50.829852 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824775    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:50.829852 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824778    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:50.829852 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824780    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:50.829852 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824783    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:50.829852 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824785    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:50.829852 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824787    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:50.829852 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824790    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:50.829852 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824792    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:50.829852 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824795    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:50.829852 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824797    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:50.829852 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824800    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:50.829852 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824802    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:50.829852 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824805    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:50.829852 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824807    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:50.829852 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824810    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:50.830338 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.824812    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:50.830338 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.824877    2583 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 17:34:50.830338 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.824907    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 17:34:50.830338 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.824915    2583 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 17:34:50.830338 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.824919    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 17:34:50.830338 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.824923    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 17:34:50.830338 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.824926    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 17:34:50.830338 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.824929    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 17:34:50.830338 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.824934    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 17:34:50.830338 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.824938    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 17:34:50.830338 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.824941    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 17:34:50.830338 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.824946    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 17:34:50.830338 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.824949    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 17:34:50.830338 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.824952    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 17:34:50.830338 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.824955    2583 flags.go:64] FLAG: --cgroup-root=""
Sep 17 17:34:50.830338 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.824958    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 17:34:50.830338 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.824961    2583 flags.go:64] FLAG: --client-ca-file=""
Sep 17 17:34:50.830338 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.824964    2583 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 17:34:50.830338 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.824967    2583 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 17:34:50.830338 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.824972    2583 flags.go:64] FLAG: --cluster-domain=""
Sep 17 17:34:50.830338 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.824975    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 17:34:50.830338 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.824978    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 17:34:50.830338 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.824981    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 17:34:50.830338 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.824984    2583 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 17:34:50.830989 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.824988    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 17:34:50.830989 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.824991    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 17:34:50.830989 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.824994    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 17:34:50.830989 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.824997    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 17:34:50.830989 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825001    2583 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 17:34:50.830989 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825004    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 17:34:50.830989 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825007    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 17:34:50.830989 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825010    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 17:34:50.830989 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825013    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 17:34:50.830989 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825017    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 17:34:50.830989 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825020    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 17:34:50.830989 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825023    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 17:34:50.830989 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825026    2583 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 17:34:50.830989 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825029    2583 flags.go:64] FLAG: --enable-server="true"
Sep 17 17:34:50.830989 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825032    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 17:34:50.830989 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825036    2583 flags.go:64] FLAG: --event-burst="100"
Sep 17 17:34:50.830989 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825039    2583 flags.go:64] FLAG: --event-qps="50"
Sep 17 17:34:50.830989 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825042    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 17:34:50.830989 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825045    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 17:34:50.830989 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825050    2583 flags.go:64] FLAG: --eviction-hard=""
Sep 17 17:34:50.830989 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825053    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 17:34:50.830989 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825058    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 17:34:50.830989 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825061    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 17:34:50.830989 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825064    2583 flags.go:64] FLAG: --eviction-soft=""
Sep 17 17:34:50.830989 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825066    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 17:34:50.831584 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825069    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 17:34:50.831584 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825072    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 17:34:50.831584 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825075    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 17:34:50.831584 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825078    2583 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 17:34:50.831584 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825081    2583 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 17:34:50.831584 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825084    2583 flags.go:64] FLAG: --feature-gates=""
Sep 17 17:34:50.831584 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825088    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 17:34:50.831584 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825090    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 17:34:50.831584 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825094    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 17:34:50.831584 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825097    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 17:34:50.831584 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825100    2583 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 17:34:50.831584 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825103    2583 flags.go:64] FLAG: --help="false"
Sep 17 17:34:50.831584 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825106    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-141-30.ec2.internal"
Sep 17 17:34:50.831584 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825109    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 17:34:50.831584 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825112    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 17:34:50.831584 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825115    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 17:34:50.831584 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825119    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 17:34:50.831584 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825122    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 17:34:50.831584 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825125    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 17:34:50.831584 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825128    2583 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 17:34:50.831584 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825131    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 17:34:50.831584 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825134    2583 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 17:34:50.831584 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825137    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 17:34:50.832181 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825140    2583 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 17:34:50.832181 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825142    2583 flags.go:64] FLAG: --kube-reserved=""
Sep 17 17:34:50.832181 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825145    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 17:34:50.832181 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825150    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 17:34:50.832181 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825153    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 17:34:50.832181 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825157    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 17:34:50.832181 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825160    2583 flags.go:64] FLAG: --lock-file=""
Sep 17 17:34:50.832181 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825163    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 17:34:50.832181 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825166    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 17:34:50.832181 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825169    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 17:34:50.832181 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825174    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 17:34:50.832181 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825177    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 17:34:50.832181 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825180    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 17:34:50.832181 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825183    2583 flags.go:64] FLAG: --logging-format="text"
Sep 17 17:34:50.832181 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825186    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 17:34:50.832181 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825190    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 17:34:50.832181 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825192    2583 flags.go:64] FLAG: --manifest-url=""
Sep 17 17:34:50.832181 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825195    2583 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 17:34:50.832181 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825199    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 17:34:50.832181 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825202    2583 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 17:34:50.832181 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825206    2583 flags.go:64] FLAG: --max-pods="110"
Sep 17 17:34:50.832181 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825209    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 17:34:50.832181 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825212    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 17:34:50.832181 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825215    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 17:34:50.832181 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825218    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 17:34:50.832791 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825221    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 17:34:50.832791 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825224    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 17:34:50.832791 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825227    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 17:34:50.832791 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825234    2583 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 17:34:50.832791 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825237    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 17:34:50.832791 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825241    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 17:34:50.832791 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825244    2583 flags.go:64] FLAG: --pod-cidr=""
Sep 17 17:34:50.832791 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825247    2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 17:34:50.832791 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825252    2583 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 17:34:50.832791 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825256    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 17:34:50.832791 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825260    2583 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 17:34:50.832791 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825264    2583 flags.go:64] FLAG: --port="10250"
Sep 17 17:34:50.832791 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825267    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 17:34:50.832791 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825272    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09fc0751ef7b768e6"
Sep 17 17:34:50.832791 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825275    2583 flags.go:64] FLAG: --qos-reserved=""
Sep 17 17:34:50.832791 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825279    2583 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 17:34:50.832791 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825282    2583 flags.go:64] FLAG: --register-node="true"
Sep 17 17:34:50.832791 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825285    2583 flags.go:64] FLAG: --register-with-taints=""
Sep 17 17:34:50.832791 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825288    2583 flags.go:64] FLAG: --registry-burst="10"
Sep 17 17:34:50.832791 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825291    2583 flags.go:64] FLAG: --registry-qps="5"
Sep 17 17:34:50.832791 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825294    2583 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 17:34:50.832791 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825296    2583 flags.go:64] FLAG: --reserved-memory=""
Sep 17 17:34:50.832791 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825300    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 17:34:50.832791 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825303    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 17:34:50.832791 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825306    2583 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 17:34:50.833406 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825309    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 17:34:50.833406 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825312    2583 flags.go:64] FLAG: --runonce="false"
Sep 17 17:34:50.833406 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825315    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 17:34:50.833406 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825318    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 17:34:50.833406 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825321    2583 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 17:34:50.833406 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825324    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 17:34:50.833406 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825327    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 17:34:50.833406 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825330    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 17:34:50.833406 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825334    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 17:34:50.833406 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825337    2583 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 17:34:50.833406 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825340    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 17:34:50.833406 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825343    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 17:34:50.833406 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825345    2583 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 17:34:50.833406 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825348    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 17:34:50.833406 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825351    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 17:34:50.833406 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825355    2583 flags.go:64] FLAG: --system-cgroups=""
Sep 17 17:34:50.833406 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825358    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 17:34:50.833406 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825364    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 17:34:50.833406 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825367    2583 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 17:34:50.833406 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825370    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 17:34:50.833406 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825376    2583 flags.go:64] FLAG: --tls-min-version=""
Sep 17 17:34:50.833406 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825379    2583 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 17:34:50.833406 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825382    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 17:34:50.833406 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825384    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 17:34:50.833406 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825387    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 17:34:50.834013 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825390    2583 flags.go:64] FLAG: --v="2"
Sep 17 17:34:50.834013 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825394    2583 flags.go:64] FLAG: --version="false"
Sep 17 17:34:50.834013 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825398    2583 flags.go:64] FLAG: --vmodule=""
Sep 17 17:34:50.834013 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825401    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 17:34:50.834013 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.825405    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 17:34:50.834013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825511    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:50.834013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825515    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:50.834013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825519    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:50.834013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825522    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:50.834013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825525    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:50.834013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825528    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:50.834013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825531    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:50.834013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825542    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:50.834013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825545    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:50.834013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825548    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:50.834013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825550    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:50.834013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825553    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:50.834013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825556    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:50.834013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825559    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:50.834013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825561    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:50.834013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825564    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:50.834567 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825567    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:50.834567 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825569    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:50.834567 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825572    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:50.834567 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825576    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:50.834567 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825579    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:50.834567 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825581    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:50.834567 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825586    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:50.834567 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825589    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:50.834567 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825592    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:50.834567 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825594    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:50.834567 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825597    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:50.834567 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825599    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:50.834567 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825602    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:50.834567 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825605    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:50.834567 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825607    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:50.834567 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825610    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:50.834567 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825613    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:50.834567 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825616    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:50.834567 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825618    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:50.834567 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825621    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:50.835072 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825623    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:50.835072 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825626    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:50.835072 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825629    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:50.835072 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825632    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:50.835072 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825635    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:50.835072 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825637    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:50.835072 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825640    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:50.835072 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825642    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:50.835072 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825645    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:50.835072 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825648    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:50.835072 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825650    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:50.835072 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825653    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:50.835072 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825670    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:50.835072 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825677    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:50.835072 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825682    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:50.835072 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825688    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:50.835072 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825692    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:50.835072 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825694    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:50.835072 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825699    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:50.835541 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825707    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:50.835541 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825710    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:50.835541 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825713    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:50.835541 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825716    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:50.835541 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825718    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:50.835541 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825722    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:50.835541 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825726    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:50.835541 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825730    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:50.835541 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825733    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:50.835541 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825736    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:50.835541 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825739    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:50.835541 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825742    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:50.835541 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825745    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:50.835541 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825748    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:50.835541 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825751    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:50.835541 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825754    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:50.835541 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825757    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:50.835541 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825759    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:50.835992 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825763    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:50.835992 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825767    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:50.835992 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825770    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:50.835992 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825773    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:50.835992 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825775    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:50.835992 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825778    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:50.835992 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825781    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:50.835992 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825783    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:50.835992 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825786    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:50.835992 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825789    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:50.835992 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825793    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:50.835992 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825796    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:50.835992 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825799    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:50.835992 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825802    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:50.835992 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825805    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:50.835992 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825808    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:50.835992 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825811    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:50.835992 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825814    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:50.835992 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825817    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:50.836469 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825820    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:50.836469 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825822    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:50.836469 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825825    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:50.836469 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825828    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:50.836469 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825830    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:50.836469 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825833    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:50.836469 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.825836    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:50.836469 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.826259    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:34:50.836469 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.833912    2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 17:34:50.836469 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.833930    2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 17:34:50.836469 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.833949    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 17:34:50.836469 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.833955    2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 17:34:50.836469 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834024    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:50.836469 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834029    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:50.836469 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834033    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:50.836864 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834036    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:50.836864 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834040    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:50.836864 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834043    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:50.836864 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834046    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:50.836864 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834049    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:50.836864 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834052    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:50.836864 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834054    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:50.836864 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834057    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:50.836864 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834060    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:50.836864 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834062    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:50.836864 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834065    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:50.836864 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834068    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:50.836864 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834071    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:50.836864 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834073    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:50.836864 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834076    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:50.836864 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834079    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:50.836864 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834081    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:50.836864 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834084    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:50.836864 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834087    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:50.837323 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834090    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:50.837323 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834095    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:50.837323 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834098    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:50.837323 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834101    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:50.837323 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834103    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:50.837323 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834106    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:50.837323 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834109    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:50.837323 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834118    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:50.837323 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834121    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:50.837323 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834123    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:50.837323 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834126    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:50.837323 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834129    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:50.837323 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834131    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:50.837323 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834134    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:50.837323 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834137    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:50.837323 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834139    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:50.837323 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834142    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:50.837323 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834144    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:50.837323 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834147    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:50.837832 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834150    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:50.837832 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834152    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:50.837832 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834155    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:50.837832 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834159    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:50.837832 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834162    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:50.837832 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834166    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:50.837832 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834169    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:50.837832 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834172    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:50.837832 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834175    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:50.837832 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834178    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:50.837832 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834182    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:50.837832 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834185    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:50.837832 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834188    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:50.837832 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834191    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:50.837832 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834193    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:50.837832 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834196    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:50.837832 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834199    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:50.837832 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834202    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:50.838283 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834204    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:50.838283 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834207    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:50.838283 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834210    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:50.838283 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834219    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:50.838283 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834221    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:50.838283 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834224    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:50.838283 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834226    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:50.838283 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834229    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:50.838283 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834232    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:50.838283 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834234    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:50.838283 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834237    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:50.838283 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834239    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:50.838283 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834242    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:50.838283 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834244    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:50.838283 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834247    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:50.838283 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834250    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:50.838283 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834252    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:50.838283 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834254    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:50.838283 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834257    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:50.838283 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834260    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:50.838283 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834262    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:50.838896 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834265    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:50.838896 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834267    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:50.838896 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834270    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:50.838896 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834273    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:50.838896 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834275    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:50.838896 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834278    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:50.838896 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834280    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:50.838896 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834283    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:50.838896 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834285    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:50.838896 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834288    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:50.838896 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834290    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:50.838896 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834293    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:50.838896 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834295    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:50.838896 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834298    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:50.838896 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834301    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:50.838896 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834309    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:50.838896 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834312    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:50.838896 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834315    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:50.838896 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834317    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:50.839374 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.834322    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:34:50.839374 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834482    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:50.839374 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834487    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:50.839374 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834490    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:50.839374 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834493    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:50.839374 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834496    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:50.839374 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834499    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:50.839374 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834501    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:50.839374 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834504    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:50.839374 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834507    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:50.839374 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834509    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:50.839374 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834512    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:50.839374 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834514    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:50.839374 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834517    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:50.839374 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834520    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:50.840013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834523    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:50.840013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834525    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:50.840013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834528    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:50.840013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834530    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:50.840013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834533    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:50.840013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834535    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:50.840013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834538    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:50.840013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834540    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:50.840013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834543    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:50.840013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834545    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:50.840013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834548    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:50.840013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834550    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:50.840013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834553    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:50.840013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834556    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:50.840013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834564    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:50.840013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834567    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:50.840013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834570    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:50.840013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834573    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:50.840013 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834575    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:50.840784 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834579    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:50.840784 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834583    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:50.840784 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834586    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:50.840784 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834589    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:50.840784 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834592    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:50.840784 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834594    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:50.840784 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834597    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:50.840784 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834599    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:50.840784 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834602    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:50.840784 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834604    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:50.840784 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834607    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:50.840784 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834610    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:50.840784 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834612    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:50.840784 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834615    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:50.840784 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834617    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:50.840784 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834620    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:50.840784 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834622    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:50.840784 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834625    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:50.840784 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834628    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:50.841325 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834630    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:50.841325 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834633    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:50.841325 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834635    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:50.841325 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834638    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:50.841325 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834641    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:50.841325 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834643    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:50.841325 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834646    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:50.841325 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834648    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:50.841325 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834651    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:50.841325 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834675    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:50.841325 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834679    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:50.841325 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834682    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:50.841325 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834684    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:50.841325 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834687    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:50.841325 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834690    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:50.841325 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834692    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:50.841325 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834695    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:50.841325 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834697    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:50.841325 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834700    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:50.841325 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834703    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:50.841835 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834705    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:50.841835 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834708    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:50.841835 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834711    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:50.841835 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834713    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:50.841835 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834716    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:50.841835 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834718    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:50.841835 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834721    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:50.841835 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834724    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:50.841835 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834726    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:50.841835 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834729    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:50.841835 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834731    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:50.841835 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834734    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:50.841835 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834736    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:50.841835 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834739    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:50.841835 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834741    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:50.841835 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834744    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:50.841835 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834747    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:50.841835 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834749    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:50.841835 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834752    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:50.842293 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834756    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:50.842293 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834759    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:50.842293 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834762    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:50.842293 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834771    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:50.842293 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834774    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:50.842293 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834776    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:50.842293 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834779    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:50.842293 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.834781    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:50.842293 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.834786    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:34:50.842293 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.834907    2583 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 17:34:50.842293 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.836790    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 17:34:50.842293 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.837457    2583 server.go:1019] "Starting client certificate rotation"
Sep 17 17:34:50.842293 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.837552    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:34:50.843910 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.837591    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:34:50.843910 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.840630    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:34:50.843910 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.842853    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:34:50.847411 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.847395    2583 log.go:25] "Validated CRI v1 runtime API"
Sep 17 17:34:50.851993 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.851978    2583 log.go:25] "Validated CRI v1 image API"
Sep 17 17:34:50.852831 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.852817    2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 17:34:50.854966 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.854948    2583 fs.go:135] Filesystem UUIDs: map[35ad9a5a-5237-4b96-866e-51fffe0441ff:/dev/nvme0n1p4 795dd697-e5c4-4b27-9408-3f19db354546:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 17 17:34:50.855027 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.854965    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 17:34:50.859695 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.859571    2583 manager.go:216] Machine: {Timestamp:2026-09-17 17:34:50.858176953 +0000 UTC m=+1.949483730 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106614 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c9363f82a0b79614904439514b9a2 SystemUUID:ec2c9363-f82a-0b79-6149-04439514b9a2 BootID:c9de11b9-052e-4fe1-9ea1-ae5886cb3851 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:db:24:0a:83 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:db:24:0a:83 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:1a:cb:70:ff:51 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 17:34:50.859695 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.859686    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 17:34:50.859835 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.859757    2583 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 17:34:50.861078 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.861057    2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 17:34:50.861213 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.861079    2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-30.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 17:34:50.862173 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.862163    2583 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 17:34:50.862206 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.862175    2583 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 17:34:50.862206 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.862183    2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:34:50.862206 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.862193    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:34:50.862291 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.862260    2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 17:34:50.865021 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.865008    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:34:50.865129 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.865119    2583 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 17:34:50.865161 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.865152    2583 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 17:34:50.865189 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.865164    2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 17:34:50.865189 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.865174    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 17:34:50.865189 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.865183    2583 kubelet.go:398] "Adding apiserver pod source"
Sep 17 17:34:50.865295 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.865200    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 17:34:50.865528 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.865513    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:34:50.865592 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.865535    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:34:50.875936 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.875910    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:34:50.877397 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.877377    2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 17:34:50.879570 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.879551    2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 17:34:50.879674 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.879592    2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 17:34:50.880417 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.880403    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 17:34:50.880498 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.880422    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 17:34:50.880498 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.880430    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 17:34:50.880498 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.880437    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 17:34:50.880498 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.880459    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 17:34:50.880498 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.880468    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 17:34:50.880498 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.880487    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 17:34:50.880498 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.880495    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 17:34:50.880757 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.880505    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 17:34:50.880757 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.880514    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 17:34:50.880757 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.880527    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 17:34:50.880757 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.880535    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 17:34:50.880757 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.880568    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 17:34:50.880757 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.880576    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 17:34:50.881254 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:50.881228    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-30.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 17:34:50.881350 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:50.881297    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 17:34:50.884443 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.884425    2583 server.go:1268] "Started kubelet"
Sep 17 17:34:50.884541 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.884465    2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 17:34:50.884589 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.884530    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 17:34:50.884589 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.884582    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 17:34:50.885264 ip-10-0-141-30 systemd[1]: Started Kubernetes Kubelet.
Sep 17 17:34:50.885774 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.885529    2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 17:34:50.885774 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.885725    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-30.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 17:34:50.885774 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.885738    2583 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 17:34:50.889115 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:50.889100    2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 17:34:50.892469 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.892453    2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 17:34:50.892544 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.892478    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 17:34:50.892544 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:50.892531    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-30.ec2.internal\" not found"
Sep 17 17:34:50.893002 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.892986    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 17:34:50.893092 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.893022    2583 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 17:34:50.893092 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.893040    2583 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 17:34:50.893198 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.893125    2583 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 17:34:50.893198 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.893136    2583 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 17:34:50.893295 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.893273    2583 factory.go:55] Registering systemd factory
Sep 17 17:34:50.893295 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.893289    2583 factory.go:223] Registration of the systemd container factory successfully
Sep 17 17:34:50.893467 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.893454    2583 factory.go:153] Registering CRI-O factory
Sep 17 17:34:50.893467 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.893467    2583 factory.go:223] Registration of the crio container factory successfully
Sep 17 17:34:50.893567 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.893557    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 17:34:50.893610 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.893575    2583 factory.go:103] Registering Raw factory
Sep 17 17:34:50.893651 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.893611    2583 manager.go:1196] Started watching for new ooms in manager
Sep 17 17:34:50.894182 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.894114    2583 manager.go:349] Starting recovery of all containers
Sep 17 17:34:50.896000 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:50.895970    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 17:34:50.896113 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:50.896092    2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-30.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 17:34:50.912357 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:50.912313    2583 helpers.go:245] readString: Failed to read "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service/memory.min": read /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service/memory.min: no such device
Sep 17 17:34:50.913243 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.913220    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6rbth"
Sep 17 17:34:50.914123 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.914074    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 17:34:50.914810 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.914790    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6rbth"
Sep 17 17:34:50.918323 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.918301    2583 manager.go:354] Recovery completed
Sep 17 17:34:50.920227 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:50.919543    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-30.ec2.internal.18d62c30ff9382d7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-30.ec2.internal,UID:ip-10-0-141-30.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-30.ec2.internal,},FirstTimestamp:2026-09-17 17:34:50.884260567 +0000 UTC m=+1.975567323,LastTimestamp:2026-09-17 17:34:50.884260567 +0000 UTC m=+1.975567323,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-30.ec2.internal,}"
Sep 17 17:34:50.922404 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.922389    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:50.924222 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.924206    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-30.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:50.924280 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.924233    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-30.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:50.924280 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.924243    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-30.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:50.924825 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.924804    2583 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 17:34:50.924825 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.924820    2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 17:34:50.924930 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.924839    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:34:50.924930 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.924852    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:50.925512 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.925498    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-30.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:50.925574 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.925524    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-30.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:50.925574 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.925538    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-30.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:50.925855 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.925839    2583 policy_none.go:49] "None policy: Start"
Sep 17 17:34:50.925855 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.925854    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 17:34:50.925962 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.925864    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 17:34:50.926830 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.926819    2583 policy_none.go:47] "Start"
Sep 17 17:34:50.974200 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.974187    2583 manager.go:337] "Starting Device Plugin manager"
Sep 17 17:34:50.975087 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:50.974220    2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 17:34:50.975087 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.974229    2583 server.go:87] "Starting device plugin registration server"
Sep 17 17:34:50.975087 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.974442    2583 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 17:34:50.975087 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.974464    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 17:34:50.975087 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.974542    2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 17:34:50.975087 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.974614    2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 17:34:50.975087 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:50.974622    2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 17:34:50.975329 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:50.975148    2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 17:34:50.975329 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:50.975190    2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-30.ec2.internal\" not found"
Sep 17 17:34:51.034204 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.034179    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 17:34:51.034285 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.034224    2583 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 17:34:51.034285 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.034246    2583 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 17:34:51.034285 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:51.034278    2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 17:34:51.036830 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.036813    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:51.075052 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.075010    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:51.077368 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.077346    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-30.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:51.077471 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.077385    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-30.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:51.077471 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.077400    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-30.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:51.077471 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.077421    2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-30.ec2.internal"
Sep 17 17:34:51.083713 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.083697    2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-30.ec2.internal"
Sep 17 17:34:51.083784 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:51.083724    2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-30.ec2.internal\": node \"ip-10-0-141-30.ec2.internal\" not found"
Sep 17 17:34:51.099393 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:51.099378    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-30.ec2.internal\" not found"
Sep 17 17:34:51.134544 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.134525    2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-30.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-30.ec2.internal"]
Sep 17 17:34:51.134600 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.134591    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:51.135394 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.135379    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-30.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:51.135437 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.135405    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-30.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:51.135437 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.135414    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-30.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:51.136455 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.136443    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:51.136566 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.136552    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-30.ec2.internal"
Sep 17 17:34:51.136613 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.136580    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:51.137250 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.137232    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-30.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:51.137329 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.137260    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-30.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:51.137329 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.137272    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-30.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:51.137329 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.137232    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-30.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:51.137329 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.137326    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-30.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:51.137449 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.137339    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-30.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:51.138375 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.138361    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-30.ec2.internal"
Sep 17 17:34:51.138456 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.138382    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:51.139114 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.139089    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-30.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:51.139195 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.139121    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-30.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:51.139195 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.139130    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-30.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:51.166318 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:51.166298    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-30.ec2.internal\" not found" node="ip-10-0-141-30.ec2.internal"
Sep 17 17:34:51.170745 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:51.170729    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-30.ec2.internal\" not found" node="ip-10-0-141-30.ec2.internal"
Sep 17 17:34:51.194876 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.194857    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5da666044a5091ad00b137b36d5ff04c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-30.ec2.internal\" (UID: \"5da666044a5091ad00b137b36d5ff04c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-30.ec2.internal"
Sep 17 17:34:51.194954 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.194880    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5da666044a5091ad00b137b36d5ff04c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-30.ec2.internal\" (UID: \"5da666044a5091ad00b137b36d5ff04c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-30.ec2.internal"
Sep 17 17:34:51.194954 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.194895    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/710fcb8c6a77db50f998ce1b3ff8fde1-config\") pod \"kube-apiserver-proxy-ip-10-0-141-30.ec2.internal\" (UID: \"710fcb8c6a77db50f998ce1b3ff8fde1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-30.ec2.internal"
Sep 17 17:34:51.199941 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:51.199922    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-30.ec2.internal\" not found"
Sep 17 17:34:51.295008 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.294991    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5da666044a5091ad00b137b36d5ff04c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-30.ec2.internal\" (UID: \"5da666044a5091ad00b137b36d5ff04c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-30.ec2.internal"
Sep 17 17:34:51.295114 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.295017    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5da666044a5091ad00b137b36d5ff04c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-30.ec2.internal\" (UID: \"5da666044a5091ad00b137b36d5ff04c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-30.ec2.internal"
Sep 17 17:34:51.295114 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.295034    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/710fcb8c6a77db50f998ce1b3ff8fde1-config\") pod \"kube-apiserver-proxy-ip-10-0-141-30.ec2.internal\" (UID: \"710fcb8c6a77db50f998ce1b3ff8fde1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-30.ec2.internal"
Sep 17 17:34:51.295114 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.295058    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/710fcb8c6a77db50f998ce1b3ff8fde1-config\") pod \"kube-apiserver-proxy-ip-10-0-141-30.ec2.internal\" (UID: \"710fcb8c6a77db50f998ce1b3ff8fde1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-30.ec2.internal"
Sep 17 17:34:51.295114 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.295074    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5da666044a5091ad00b137b36d5ff04c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-30.ec2.internal\" (UID: \"5da666044a5091ad00b137b36d5ff04c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-30.ec2.internal"
Sep 17 17:34:51.295114 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.295093    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5da666044a5091ad00b137b36d5ff04c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-30.ec2.internal\" (UID: \"5da666044a5091ad00b137b36d5ff04c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-30.ec2.internal"
Sep 17 17:34:51.300108 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:51.300095    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-30.ec2.internal\" not found"
Sep 17 17:34:51.400878 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:51.400825    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-30.ec2.internal\" not found"
Sep 17 17:34:51.470024 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.469992    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-30.ec2.internal"
Sep 17 17:34:51.473894 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.473701    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-30.ec2.internal"
Sep 17 17:34:51.501186 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:51.501163    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-30.ec2.internal\" not found"
Sep 17 17:34:51.601672 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:51.601636    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-30.ec2.internal\" not found"
Sep 17 17:34:51.702208 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:51.702148    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-30.ec2.internal\" not found"
Sep 17 17:34:51.802761 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:51.802730    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-30.ec2.internal\" not found"
Sep 17 17:34:51.837907 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.837881    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 17:34:51.838292 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.838027    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:34:51.903715 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:51.903690    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-30.ec2.internal\" not found"
Sep 17 17:34:51.919387 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.919352    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 17:29:50 +0000 UTC" deadline="2028-03-07 16:12:51.69895093 +0000 UTC"
Sep 17 17:34:51.919387 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.919384    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12886h37m59.779571127s"
Sep 17 17:34:51.979790 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.979728    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:51.992891 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:51.992870    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-30.ec2.internal"
Sep 17 17:34:52.003804 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.003782    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:34:52.005225 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.005211    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-30.ec2.internal"
Sep 17 17:34:52.016907 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.016893    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:34:52.070209 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:52.070185    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod710fcb8c6a77db50f998ce1b3ff8fde1.slice/crio-9e260962948a5ec73d99e0d5407940e4ff91d4bd21ae1aec6d45b1f7c4b7e560 WatchSource:0}: Error finding container 9e260962948a5ec73d99e0d5407940e4ff91d4bd21ae1aec6d45b1f7c4b7e560: Status 404 returned error can't find the container with id 9e260962948a5ec73d99e0d5407940e4ff91d4bd21ae1aec6d45b1f7c4b7e560
Sep 17 17:34:52.072433 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:52.072402    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5da666044a5091ad00b137b36d5ff04c.slice/crio-94e679826b76d487888e84846cafa11907aa47c0b7d8a686f4e16b276e20f596 WatchSource:0}: Error finding container 94e679826b76d487888e84846cafa11907aa47c0b7d8a686f4e16b276e20f596: Status 404 returned error can't find the container with id 94e679826b76d487888e84846cafa11907aa47c0b7d8a686f4e16b276e20f596
Sep 17 17:34:52.076784 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.076772    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:34:52.374157 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.374075    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:52.458695 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.458669    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:52.866938 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.866855    2583 apiserver.go:52] "Watching apiserver"
Sep 17 17:34:52.873035 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.873006    2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 17:34:52.873413 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.873385    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-30.ec2.internal","openshift-multus/multus-additional-cni-plugins-7rbtw","openshift-multus/multus-gnhzr","openshift-multus/network-metrics-daemon-fh98p","kube-system/global-pull-secret-syncer-c46w4","kube-system/konnectivity-agent-gtn5h","kube-system/kube-apiserver-proxy-ip-10-0-141-30.ec2.internal","openshift-network-diagnostics/network-check-target-ksb2q","openshift-network-operator/iptables-alerter-9tbc5","openshift-ovn-kubernetes/ovnkube-node-9fhwv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n586","openshift-cluster-node-tuning-operator/tuned-h8287","openshift-dns/node-resolver-gzt58","openshift-image-registry/node-ca-ph456"]
Sep 17 17:34:52.876024 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.875999    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gzt58"
Sep 17 17:34:52.878205 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.878054    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:52.879337 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.879316    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-g7r8f\""
Sep 17 17:34:52.880334 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.880312    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh98p"
Sep 17 17:34:52.880448 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:52.880381    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh98p" podUID="ae10190f-448c-4e69-a268-85f77c85d7b6"
Sep 17 17:34:52.882568 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.882549    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:34:52.882702 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:52.882613    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c46w4" podUID="4696b495-138f-4b5a-838b-d13ec3d8a3a6"
Sep 17 17:34:52.883866 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.883480    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 17:34:52.883866 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.883820    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 17:34:52.883866 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.883820    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 17:34:52.884281 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.884245    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zl5dp\""
Sep 17 17:34:52.884601 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.884576    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 17:34:52.885050 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.884941    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gtn5h"
Sep 17 17:34:52.885136 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.885087    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 17:34:52.885293 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.885276    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 17:34:52.888611 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.888596    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9tbc5"
Sep 17 17:34:52.890558 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.890529    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 17:34:52.890717 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.890699    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ph456"
Sep 17 17:34:52.891847 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.891081    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 17:34:52.891847 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.891116    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-k4dp7\""
Sep 17 17:34:52.893887 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.893475    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 17:34:52.893887 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.893828    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7rbtw"
Sep 17 17:34:52.897135 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.897111    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 17:34:52.897238 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.897202    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2rwpn\""
Sep 17 17:34:52.897296 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.897277    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:34:52.897998 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.897626    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 17:34:52.897998 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.897652    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 17:34:52.897998 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.897738    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-k2ldr\""
Sep 17 17:34:52.897998 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.897676    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 17:34:52.897998 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.897913    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 17:34:52.897998 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.897936    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:52.897998 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.897979    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 17:34:52.900227 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.900207    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n586"
Sep 17 17:34:52.902342 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.902319    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-run-ovn\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:52.902430 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.902354    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b92cf805-a323-422f-b021-6e8d9bce0099-tmp-dir\") pod \"node-resolver-gzt58\" (UID: \"b92cf805-a323-422f-b021-6e8d9bce0099\") " pod="openshift-dns/node-resolver-gzt58"
Sep 17 17:34:52.902430 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.902381    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-host-var-lib-cni-bin\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:52.902430 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.902405    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ebc5fcd3-7a3c-4922-b880-c677bdb77777-cnibin\") pod \"multus-additional-cni-plugins-7rbtw\" (UID: \"ebc5fcd3-7a3c-4922-b880-c677bdb77777\") " pod="openshift-multus/multus-additional-cni-plugins-7rbtw"
Sep 17 17:34:52.902590 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.902429    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9llwv\" (UniqueName: \"kubernetes.io/projected/57984009-dbb5-41b6-a1cb-d66608916495-kube-api-access-9llwv\") pod \"iptables-alerter-9tbc5\" (UID: \"57984009-dbb5-41b6-a1cb-d66608916495\") " pod="openshift-network-operator/iptables-alerter-9tbc5"
Sep 17 17:34:52.902590 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.902466    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-host-cni-netd\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:52.902590 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.902488    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae10190f-448c-4e69-a268-85f77c85d7b6-metrics-certs\") pod \"network-metrics-daemon-fh98p\" (UID: \"ae10190f-448c-4e69-a268-85f77c85d7b6\") " pod="openshift-multus/network-metrics-daemon-fh98p"
Sep 17 17:34:52.902590 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.902511    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsw9r\" (UniqueName: \"kubernetes.io/projected/a20d51c6-5fef-4251-9e8f-ee4318f97741-kube-api-access-xsw9r\") pod \"node-ca-ph456\" (UID: \"a20d51c6-5fef-4251-9e8f-ee4318f97741\") " pod="openshift-image-registry/node-ca-ph456"
Sep 17 17:34:52.902590 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.902537    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f5a84a2-2e30-413a-823f-29cbf761cee0-ovn-node-metrics-cert\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:52.902590 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.902559    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-system-cni-dir\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:52.902590 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.902582    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3541d447-ae55-428c-9ab4-620f06f9c2fa-cni-binary-copy\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:52.902926 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.902606    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cflz\" (UniqueName: \"kubernetes.io/projected/3541d447-ae55-428c-9ab4-620f06f9c2fa-kube-api-access-2cflz\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:52.902926 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.902736    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f5a84a2-2e30-413a-823f-29cbf761cee0-ovnkube-script-lib\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:52.902926 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.902745    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:52.902926 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.902769    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-var-lib-openvswitch\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:52.902926 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.902818    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-host-kubelet\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:52.902926 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.902855    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-run-openvswitch\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:52.902926 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.902883    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gqg9\" (UniqueName: \"kubernetes.io/projected/ebc5fcd3-7a3c-4922-b880-c677bdb77777-kube-api-access-7gqg9\") pod \"multus-additional-cni-plugins-7rbtw\" (UID: \"ebc5fcd3-7a3c-4922-b880-c677bdb77777\") " pod="openshift-multus/multus-additional-cni-plugins-7rbtw"
Sep 17 17:34:52.902926 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.902909    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/57984009-dbb5-41b6-a1cb-d66608916495-iptables-alerter-script\") pod \"iptables-alerter-9tbc5\" (UID: \"57984009-dbb5-41b6-a1cb-d66608916495\") " pod="openshift-network-operator/iptables-alerter-9tbc5"
Sep 17 17:34:52.903295 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.902931    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-systemd-units\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:52.903295 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.902954    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a20d51c6-5fef-4251-9e8f-ee4318f97741-serviceca\") pod \"node-ca-ph456\" (UID: \"a20d51c6-5fef-4251-9e8f-ee4318f97741\") " pod="openshift-image-registry/node-ca-ph456"
Sep 17 17:34:52.903295 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.902979    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ebc5fcd3-7a3c-4922-b880-c677bdb77777-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7rbtw\" (UID: \"ebc5fcd3-7a3c-4922-b880-c677bdb77777\") " pod="openshift-multus/multus-additional-cni-plugins-7rbtw"
Sep 17 17:34:52.903295 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.903000    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/798a6256-4107-44bb-9da2-b3a7c769eda1-konnectivity-ca\") pod \"konnectivity-agent-gtn5h\" (UID: \"798a6256-4107-44bb-9da2-b3a7c769eda1\") " pod="kube-system/konnectivity-agent-gtn5h"
Sep 17 17:34:52.903295 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.903023    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-run-systemd\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:52.903295 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.903130    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-node-log\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:52.903295 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.903162    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-host-run-k8s-cni-cncf-io\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:52.903295 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.903188    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-multus-socket-dir-parent\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:52.903295 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.903220    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a20d51c6-5fef-4251-9e8f-ee4318f97741-host\") pod \"node-ca-ph456\" (UID: \"a20d51c6-5fef-4251-9e8f-ee4318f97741\") " pod="openshift-image-registry/node-ca-ph456"
Sep 17 17:34:52.903295 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.903245    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-multus-conf-dir\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:52.903717 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.903339    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-host-cni-bin\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:52.903717 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.903364    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-host-slash\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:52.903717 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.903389    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-host-run-ovn-kubernetes\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:52.903717 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.903414    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-cnibin\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:52.903717 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.903476    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4696b495-138f-4b5a-838b-d13ec3d8a3a6-kubelet-config\") pod \"global-pull-secret-syncer-c46w4\" (UID: \"4696b495-138f-4b5a-838b-d13ec3d8a3a6\") " pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:34:52.903717 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.903585    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4696b495-138f-4b5a-838b-d13ec3d8a3a6-original-pull-secret\") pod \"global-pull-secret-syncer-c46w4\" (UID: \"4696b495-138f-4b5a-838b-d13ec3d8a3a6\") " pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:34:52.903717 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.903614    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-etc-openvswitch\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:52.903717 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.903638    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b92cf805-a323-422f-b021-6e8d9bce0099-hosts-file\") pod \"node-resolver-gzt58\" (UID: \"b92cf805-a323-422f-b021-6e8d9bce0099\") " pod="openshift-dns/node-resolver-gzt58"
Sep 17 17:34:52.904028 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.903757    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-multus-cni-dir\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:52.904028 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.903792    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v45lp\" (UniqueName: \"kubernetes.io/projected/ae10190f-448c-4e69-a268-85f77c85d7b6-kube-api-access-v45lp\") pod \"network-metrics-daemon-fh98p\" (UID: \"ae10190f-448c-4e69-a268-85f77c85d7b6\") " pod="openshift-multus/network-metrics-daemon-fh98p"
Sep 17 17:34:52.904028 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.903817    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebc5fcd3-7a3c-4922-b880-c677bdb77777-system-cni-dir\") pod \"multus-additional-cni-plugins-7rbtw\" (UID: \"ebc5fcd3-7a3c-4922-b880-c677bdb77777\") " pod="openshift-multus/multus-additional-cni-plugins-7rbtw"
Sep 17 17:34:52.904028 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.903842    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ebc5fcd3-7a3c-4922-b880-c677bdb77777-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7rbtw\" (UID: \"ebc5fcd3-7a3c-4922-b880-c677bdb77777\") " pod="openshift-multus/multus-additional-cni-plugins-7rbtw"
Sep 17 17:34:52.904028 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.903879    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f5a84a2-2e30-413a-823f-29cbf761cee0-ovnkube-config\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:52.904028 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.903931    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-log-socket\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:52.904028 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.903965    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-host-run-netns\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:52.904028 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.903988    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/798a6256-4107-44bb-9da2-b3a7c769eda1-agent-certs\") pod \"konnectivity-agent-gtn5h\" (UID: \"798a6256-4107-44bb-9da2-b3a7c769eda1\") " pod="kube-system/konnectivity-agent-gtn5h"
Sep 17 17:34:52.904028 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.904006    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-host-var-lib-cni-multus\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:52.904028 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.904028    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-host-run-multus-certs\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:52.904359 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.904043    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4696b495-138f-4b5a-838b-d13ec3d8a3a6-dbus\") pod \"global-pull-secret-syncer-c46w4\" (UID: \"4696b495-138f-4b5a-838b-d13ec3d8a3a6\") " pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:34:52.904359 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.904067    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/57984009-dbb5-41b6-a1cb-d66608916495-host-slash\") pod \"iptables-alerter-9tbc5\" (UID: \"57984009-dbb5-41b6-a1cb-d66608916495\") " pod="openshift-network-operator/iptables-alerter-9tbc5"
Sep 17 17:34:52.904359 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.904090    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ebc5fcd3-7a3c-4922-b880-c677bdb77777-cni-binary-copy\") pod \"multus-additional-cni-plugins-7rbtw\" (UID: \"ebc5fcd3-7a3c-4922-b880-c677bdb77777\") " pod="openshift-multus/multus-additional-cni-plugins-7rbtw"
Sep 17 17:34:52.904359 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.904111    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ebc5fcd3-7a3c-4922-b880-c677bdb77777-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7rbtw\" (UID: \"ebc5fcd3-7a3c-4922-b880-c677bdb77777\") " pod="openshift-multus/multus-additional-cni-plugins-7rbtw"
Sep 17 17:34:52.904359 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.904125    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f5a84a2-2e30-413a-823f-29cbf761cee0-env-overrides\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:52.904359 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.904158    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c226j\" (UniqueName: \"kubernetes.io/projected/b92cf805-a323-422f-b021-6e8d9bce0099-kube-api-access-c226j\") pod \"node-resolver-gzt58\" (UID: \"b92cf805-a323-422f-b021-6e8d9bce0099\") " pod="openshift-dns/node-resolver-gzt58"
Sep 17 17:34:52.904359 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.904176    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-hostroot\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:52.904359 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.904189    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3541d447-ae55-428c-9ab4-620f06f9c2fa-multus-daemon-config\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:52.904359 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.904204    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-etc-kubernetes\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:52.904359 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.904237    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:52.904359 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.904275    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t4jw\" (UniqueName: \"kubernetes.io/projected/6f5a84a2-2e30-413a-823f-29cbf761cee0-kube-api-access-5t4jw\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:52.904359 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.904305    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-host-run-netns\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:52.904359 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.904330    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-host-var-lib-kubelet\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:52.905554 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.905537    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksb2q"
Sep 17 17:34:52.905637 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:52.905596    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksb2q" podUID="df33b3f5-aa93-4cb0-b1cb-3d89813c9bac"
Sep 17 17:34:52.906948 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.906762    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 17:34:52.906948 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.906776    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 17:34:52.906948 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.906829    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 17:34:52.906948 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.906918    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 17:34:52.906948 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.906947    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 17:34:52.907232 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.907019    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 17:34:52.907232 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.907025    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 17:34:52.907232 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.907216    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 17:34:52.907382 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.907233    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4jdj4\""
Sep 17 17:34:52.907382 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.907260    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 17:34:52.907382 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.907271    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8nck5\""
Sep 17 17:34:52.907382 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.907359    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:34:52.907564 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.907415    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 17:34:52.907564 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.907519    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 17:34:52.907674 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.907574    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gg4fk\""
Sep 17 17:34:52.907785 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.907770    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lpklv\""
Sep 17 17:34:52.921859 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.921842    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:34:52.950323 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.950301    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cctz2"
Sep 17 17:34:52.964346 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.964327    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cctz2"
Sep 17 17:34:52.994057 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:52.994035    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 17:34:53.004767 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.004746    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f5a84a2-2e30-413a-823f-29cbf761cee0-ovn-node-metrics-cert\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.004865 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.004769    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-system-cni-dir\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:53.004865 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.004785    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3541d447-ae55-428c-9ab4-620f06f9c2fa-cni-binary-copy\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:53.004865 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.004799    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cflz\" (UniqueName: \"kubernetes.io/projected/3541d447-ae55-428c-9ab4-620f06f9c2fa-kube-api-access-2cflz\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:53.004865 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.004832    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f5a84a2-2e30-413a-823f-29cbf761cee0-ovnkube-script-lib\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.005095 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.004872    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-system-cni-dir\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:53.005285 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.005242    2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 17:34:53.005398 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.005380    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f5a84a2-2e30-413a-823f-29cbf761cee0-ovnkube-script-lib\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.005398 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.005387    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3541d447-ae55-428c-9ab4-620f06f9c2fa-cni-binary-copy\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:53.005533 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.005401    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-var-lib-openvswitch\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.005533 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.005442    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-host-kubelet\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.005533 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.005468    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-run-openvswitch\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.005533 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.005484    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-var-lib-openvswitch\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.005533 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.005493    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gqg9\" (UniqueName: \"kubernetes.io/projected/ebc5fcd3-7a3c-4922-b880-c677bdb77777-kube-api-access-7gqg9\") pod \"multus-additional-cni-plugins-7rbtw\" (UID: \"ebc5fcd3-7a3c-4922-b880-c677bdb77777\") " pod="openshift-multus/multus-additional-cni-plugins-7rbtw"
Sep 17 17:34:53.005533 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.005524    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c1593193-cc86-43b4-bfd5-75855a7d027d-sys-fs\") pod \"aws-ebs-csi-driver-node-9n586\" (UID: \"c1593193-cc86-43b4-bfd5-75855a7d027d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n586"
Sep 17 17:34:53.005533 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.005528    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-run-openvswitch\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.005901 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.005533    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-host-kubelet\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.005901 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.005547    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-etc-tuned\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.005901 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.005571    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/57984009-dbb5-41b6-a1cb-d66608916495-iptables-alerter-script\") pod \"iptables-alerter-9tbc5\" (UID: \"57984009-dbb5-41b6-a1cb-d66608916495\") " pod="openshift-network-operator/iptables-alerter-9tbc5"
Sep 17 17:34:53.005901 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.005635    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-systemd-units\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.005901 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.005678    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a20d51c6-5fef-4251-9e8f-ee4318f97741-serviceca\") pod \"node-ca-ph456\" (UID: \"a20d51c6-5fef-4251-9e8f-ee4318f97741\") " pod="openshift-image-registry/node-ca-ph456"
Sep 17 17:34:53.005901 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.005706    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ebc5fcd3-7a3c-4922-b880-c677bdb77777-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7rbtw\" (UID: \"ebc5fcd3-7a3c-4922-b880-c677bdb77777\") " pod="openshift-multus/multus-additional-cni-plugins-7rbtw"
Sep 17 17:34:53.005901 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.005735    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c1593193-cc86-43b4-bfd5-75855a7d027d-device-dir\") pod \"aws-ebs-csi-driver-node-9n586\" (UID: \"c1593193-cc86-43b4-bfd5-75855a7d027d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n586"
Sep 17 17:34:53.005901 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.005746    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-systemd-units\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.005901 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.005761    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-etc-sysctl-conf\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.005901 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.005787    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/798a6256-4107-44bb-9da2-b3a7c769eda1-konnectivity-ca\") pod \"konnectivity-agent-gtn5h\" (UID: \"798a6256-4107-44bb-9da2-b3a7c769eda1\") " pod="kube-system/konnectivity-agent-gtn5h"
Sep 17 17:34:53.005901 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.005809    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-run-systemd\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.005901 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.005832    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-node-log\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.005901 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.005855    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-host-run-k8s-cni-cncf-io\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:53.005901 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.005878    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-lib-modules\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.005901 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.005902    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-multus-socket-dir-parent\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:53.006497 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.005926    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a20d51c6-5fef-4251-9e8f-ee4318f97741-host\") pod \"node-ca-ph456\" (UID: \"a20d51c6-5fef-4251-9e8f-ee4318f97741\") " pod="openshift-image-registry/node-ca-ph456"
Sep 17 17:34:53.006497 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.005947    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-multus-conf-dir\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:53.006497 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.005967    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-host-cni-bin\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.006497 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.005988    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-host-slash\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.006497 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.006010    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-host-run-ovn-kubernetes\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.006497 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.006034    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-cnibin\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:53.006497 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.006063    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4696b495-138f-4b5a-838b-d13ec3d8a3a6-kubelet-config\") pod \"global-pull-secret-syncer-c46w4\" (UID: \"4696b495-138f-4b5a-838b-d13ec3d8a3a6\") " pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:34:53.006497 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.006088    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4696b495-138f-4b5a-838b-d13ec3d8a3a6-original-pull-secret\") pod \"global-pull-secret-syncer-c46w4\" (UID: \"4696b495-138f-4b5a-838b-d13ec3d8a3a6\") " pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:34:53.006497 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.006110    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c1593193-cc86-43b4-bfd5-75855a7d027d-etc-selinux\") pod \"aws-ebs-csi-driver-node-9n586\" (UID: \"c1593193-cc86-43b4-bfd5-75855a7d027d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n586"
Sep 17 17:34:53.006497 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.006132    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-host\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.006497 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.006138    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/57984009-dbb5-41b6-a1cb-d66608916495-iptables-alerter-script\") pod \"iptables-alerter-9tbc5\" (UID: \"57984009-dbb5-41b6-a1cb-d66608916495\") " pod="openshift-network-operator/iptables-alerter-9tbc5"
Sep 17 17:34:53.006497 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.006148    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a20d51c6-5fef-4251-9e8f-ee4318f97741-serviceca\") pod \"node-ca-ph456\" (UID: \"a20d51c6-5fef-4251-9e8f-ee4318f97741\") " pod="openshift-image-registry/node-ca-ph456"
Sep 17 17:34:53.006497 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.006155    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm2vq\" (UniqueName: \"kubernetes.io/projected/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-kube-api-access-vm2vq\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.006497 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.006178    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-etc-openvswitch\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.006497 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.006202    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-host-cni-bin\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.006497 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.006218    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-run-systemd\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.006497 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.006230    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-host-slash\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.007257 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.006247    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b92cf805-a323-422f-b021-6e8d9bce0099-hosts-file\") pod \"node-resolver-gzt58\" (UID: \"b92cf805-a323-422f-b021-6e8d9bce0099\") " pod="openshift-dns/node-resolver-gzt58"
Sep 17 17:34:53.007257 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.006271    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-multus-cni-dir\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:53.007257 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.006273    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ebc5fcd3-7a3c-4922-b880-c677bdb77777-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7rbtw\" (UID: \"ebc5fcd3-7a3c-4922-b880-c677bdb77777\") " pod="openshift-multus/multus-additional-cni-plugins-7rbtw"
Sep 17 17:34:53.007257 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.006295    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v45lp\" (UniqueName: \"kubernetes.io/projected/ae10190f-448c-4e69-a268-85f77c85d7b6-kube-api-access-v45lp\") pod \"network-metrics-daemon-fh98p\" (UID: \"ae10190f-448c-4e69-a268-85f77c85d7b6\") " pod="openshift-multus/network-metrics-daemon-fh98p"
Sep 17 17:34:53.007257 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.006316    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebc5fcd3-7a3c-4922-b880-c677bdb77777-system-cni-dir\") pod \"multus-additional-cni-plugins-7rbtw\" (UID: \"ebc5fcd3-7a3c-4922-b880-c677bdb77777\") " pod="openshift-multus/multus-additional-cni-plugins-7rbtw"
Sep 17 17:34:53.007257 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.006323    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-multus-socket-dir-parent\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:53.007257 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.006325    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4696b495-138f-4b5a-838b-d13ec3d8a3a6-kubelet-config\") pod \"global-pull-secret-syncer-c46w4\" (UID: \"4696b495-138f-4b5a-838b-d13ec3d8a3a6\") " pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:34:53.007257 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.006476    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-cnibin\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:53.007257 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.006593    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a20d51c6-5fef-4251-9e8f-ee4318f97741-host\") pod \"node-ca-ph456\" (UID: \"a20d51c6-5fef-4251-9e8f-ee4318f97741\") " pod="openshift-image-registry/node-ca-ph456"
Sep 17 17:34:53.007257 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.006277    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-host-run-ovn-kubernetes\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.007257 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:53.006757    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:53.007257 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:53.006852    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4696b495-138f-4b5a-838b-d13ec3d8a3a6-original-pull-secret podName:4696b495-138f-4b5a-838b-d13ec3d8a3a6 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:53.506814801 +0000 UTC m=+4.598121544 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4696b495-138f-4b5a-838b-d13ec3d8a3a6-original-pull-secret") pod "global-pull-secret-syncer-c46w4" (UID: "4696b495-138f-4b5a-838b-d13ec3d8a3a6") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:53.007257 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.006908    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/798a6256-4107-44bb-9da2-b3a7c769eda1-konnectivity-ca\") pod \"konnectivity-agent-gtn5h\" (UID: \"798a6256-4107-44bb-9da2-b3a7c769eda1\") " pod="kube-system/konnectivity-agent-gtn5h"
Sep 17 17:34:53.007257 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.006975    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-multus-conf-dir\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:53.007257 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.006985    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-etc-openvswitch\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.007257 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.007033    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-node-log\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.007257 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.007086    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b92cf805-a323-422f-b021-6e8d9bce0099-hosts-file\") pod \"node-resolver-gzt58\" (UID: \"b92cf805-a323-422f-b021-6e8d9bce0099\") " pod="openshift-dns/node-resolver-gzt58"
Sep 17 17:34:53.008064 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.007113    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-multus-cni-dir\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:53.008064 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.007165    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-host-run-k8s-cni-cncf-io\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:53.008064 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.007211    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ebc5fcd3-7a3c-4922-b880-c677bdb77777-system-cni-dir\") pod \"multus-additional-cni-plugins-7rbtw\" (UID: \"ebc5fcd3-7a3c-4922-b880-c677bdb77777\") " pod="openshift-multus/multus-additional-cni-plugins-7rbtw"
Sep 17 17:34:53.008064 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.007337    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ebc5fcd3-7a3c-4922-b880-c677bdb77777-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7rbtw\" (UID: \"ebc5fcd3-7a3c-4922-b880-c677bdb77777\") " pod="openshift-multus/multus-additional-cni-plugins-7rbtw"
Sep 17 17:34:53.008064 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.007380    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-etc-sysconfig\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.008064 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.007406    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-etc-kubernetes\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.008064 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.007464    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-run\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.008064 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.007502    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f5a84a2-2e30-413a-823f-29cbf761cee0-ovnkube-config\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.008064 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.007601    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-log-socket\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.008064 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.007648    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-host-run-netns\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:53.008064 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.007649    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ebc5fcd3-7a3c-4922-b880-c677bdb77777-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7rbtw\" (UID: \"ebc5fcd3-7a3c-4922-b880-c677bdb77777\") " pod="openshift-multus/multus-additional-cni-plugins-7rbtw"
Sep 17 17:34:53.008064 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.007701    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c1593193-cc86-43b4-bfd5-75855a7d027d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9n586\" (UID: \"c1593193-cc86-43b4-bfd5-75855a7d027d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n586"
Sep 17 17:34:53.008064 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.007768    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-host-run-netns\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:53.008064 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.007813    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-log-socket\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.008064 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.007852    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-etc-systemd\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.008064 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.007889    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/798a6256-4107-44bb-9da2-b3a7c769eda1-agent-certs\") pod \"konnectivity-agent-gtn5h\" (UID: \"798a6256-4107-44bb-9da2-b3a7c769eda1\") " pod="kube-system/konnectivity-agent-gtn5h"
Sep 17 17:34:53.008064 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.007942    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-host-var-lib-cni-multus\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:53.008783 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.007978    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-host-run-multus-certs\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:53.008783 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.007993    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f5a84a2-2e30-413a-823f-29cbf761cee0-ovnkube-config\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.008783 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.008011    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4696b495-138f-4b5a-838b-d13ec3d8a3a6-dbus\") pod \"global-pull-secret-syncer-c46w4\" (UID: \"4696b495-138f-4b5a-838b-d13ec3d8a3a6\") " pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:34:53.008783 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.008061    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c1593193-cc86-43b4-bfd5-75855a7d027d-socket-dir\") pod \"aws-ebs-csi-driver-node-9n586\" (UID: \"c1593193-cc86-43b4-bfd5-75855a7d027d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n586"
Sep 17 17:34:53.008783 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.008075    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4696b495-138f-4b5a-838b-d13ec3d8a3a6-dbus\") pod \"global-pull-secret-syncer-c46w4\" (UID: \"4696b495-138f-4b5a-838b-d13ec3d8a3a6\") " pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:34:53.008783 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.008150    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-host-run-multus-certs\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:53.008783 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.008195    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-host-var-lib-cni-multus\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:53.008783 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.008245    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/57984009-dbb5-41b6-a1cb-d66608916495-host-slash\") pod \"iptables-alerter-9tbc5\" (UID: \"57984009-dbb5-41b6-a1cb-d66608916495\") " pod="openshift-network-operator/iptables-alerter-9tbc5"
Sep 17 17:34:53.008783 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.008282    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ebc5fcd3-7a3c-4922-b880-c677bdb77777-cni-binary-copy\") pod \"multus-additional-cni-plugins-7rbtw\" (UID: \"ebc5fcd3-7a3c-4922-b880-c677bdb77777\") " pod="openshift-multus/multus-additional-cni-plugins-7rbtw"
Sep 17 17:34:53.008783 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.008319    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ebc5fcd3-7a3c-4922-b880-c677bdb77777-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7rbtw\" (UID: \"ebc5fcd3-7a3c-4922-b880-c677bdb77777\") " pod="openshift-multus/multus-additional-cni-plugins-7rbtw"
Sep 17 17:34:53.008783 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.008351    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q862\" (UniqueName: \"kubernetes.io/projected/c1593193-cc86-43b4-bfd5-75855a7d027d-kube-api-access-8q862\") pod \"aws-ebs-csi-driver-node-9n586\" (UID: \"c1593193-cc86-43b4-bfd5-75855a7d027d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n586"
Sep 17 17:34:53.008783 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.008383    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-etc-modprobe-d\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.008783 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.008413    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-var-lib-kubelet\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.008783 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.008443    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f5a84a2-2e30-413a-823f-29cbf761cee0-env-overrides\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.008783 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.008473    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c226j\" (UniqueName: \"kubernetes.io/projected/b92cf805-a323-422f-b021-6e8d9bce0099-kube-api-access-c226j\") pod \"node-resolver-gzt58\" (UID: \"b92cf805-a323-422f-b021-6e8d9bce0099\") " pod="openshift-dns/node-resolver-gzt58"
Sep 17 17:34:53.008783 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.008497    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-hostroot\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:53.008783 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.008524    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3541d447-ae55-428c-9ab4-620f06f9c2fa-multus-daemon-config\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:53.009499 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.008555    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c1593193-cc86-43b4-bfd5-75855a7d027d-registration-dir\") pod \"aws-ebs-csi-driver-node-9n586\" (UID: \"c1593193-cc86-43b4-bfd5-75855a7d027d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n586"
Sep 17 17:34:53.009499 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.008587    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-etc-kubernetes\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:53.009499 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.008613    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.009499 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.008642    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t4jw\" (UniqueName: \"kubernetes.io/projected/6f5a84a2-2e30-413a-823f-29cbf761cee0-kube-api-access-5t4jw\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.009499 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.008717    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-host-run-netns\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.009499 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.008745    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-host-var-lib-kubelet\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:53.009499 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.008786    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-run-ovn\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.009499 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.008815    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b92cf805-a323-422f-b021-6e8d9bce0099-tmp-dir\") pod \"node-resolver-gzt58\" (UID: \"b92cf805-a323-422f-b021-6e8d9bce0099\") " pod="openshift-dns/node-resolver-gzt58"
Sep 17 17:34:53.009499 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.008817    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ebc5fcd3-7a3c-4922-b880-c677bdb77777-cni-binary-copy\") pod \"multus-additional-cni-plugins-7rbtw\" (UID: \"ebc5fcd3-7a3c-4922-b880-c677bdb77777\") " pod="openshift-multus/multus-additional-cni-plugins-7rbtw"
Sep 17 17:34:53.009499 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.008872    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-host-var-lib-cni-bin\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:53.009499 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.008910    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ebc5fcd3-7a3c-4922-b880-c677bdb77777-cnibin\") pod \"multus-additional-cni-plugins-7rbtw\" (UID: \"ebc5fcd3-7a3c-4922-b880-c677bdb77777\") " pod="openshift-multus/multus-additional-cni-plugins-7rbtw"
Sep 17 17:34:53.009499 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.009121    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-run-ovn\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.009499 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.009128    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-host-var-lib-kubelet\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:53.009499 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.009175    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87775\" (UniqueName: \"kubernetes.io/projected/df33b3f5-aa93-4cb0-b1cb-3d89813c9bac-kube-api-access-87775\") pod \"network-check-target-ksb2q\" (UID: \"df33b3f5-aa93-4cb0-b1cb-3d89813c9bac\") " pod="openshift-network-diagnostics/network-check-target-ksb2q"
Sep 17 17:34:53.009499 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.009207    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-etc-sysctl-d\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.009499 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.009229    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-sys\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.009499 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.009283    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9llwv\" (UniqueName: \"kubernetes.io/projected/57984009-dbb5-41b6-a1cb-d66608916495-kube-api-access-9llwv\") pod \"iptables-alerter-9tbc5\" (UID: \"57984009-dbb5-41b6-a1cb-d66608916495\") " pod="openshift-network-operator/iptables-alerter-9tbc5"
Sep 17 17:34:53.010297 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.009318    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-host-cni-netd\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.010297 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.009319    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ebc5fcd3-7a3c-4922-b880-c677bdb77777-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7rbtw\" (UID: \"ebc5fcd3-7a3c-4922-b880-c677bdb77777\") " pod="openshift-multus/multus-additional-cni-plugins-7rbtw"
Sep 17 17:34:53.010297 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.009389    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae10190f-448c-4e69-a268-85f77c85d7b6-metrics-certs\") pod \"network-metrics-daemon-fh98p\" (UID: \"ae10190f-448c-4e69-a268-85f77c85d7b6\") " pod="openshift-multus/network-metrics-daemon-fh98p"
Sep 17 17:34:53.010297 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.009419    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsw9r\" (UniqueName: \"kubernetes.io/projected/a20d51c6-5fef-4251-9e8f-ee4318f97741-kube-api-access-xsw9r\") pod \"node-ca-ph456\" (UID: \"a20d51c6-5fef-4251-9e8f-ee4318f97741\") " pod="openshift-image-registry/node-ca-ph456"
Sep 17 17:34:53.010297 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.009449    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-tmp\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.010297 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.009607    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-hostroot\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:53.010297 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.009684    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-etc-kubernetes\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:53.010297 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.009729    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.010297 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.009771    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3541d447-ae55-428c-9ab4-620f06f9c2fa-host-var-lib-cni-bin\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:53.010297 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.009845    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/57984009-dbb5-41b6-a1cb-d66608916495-host-slash\") pod \"iptables-alerter-9tbc5\" (UID: \"57984009-dbb5-41b6-a1cb-d66608916495\") " pod="openshift-network-operator/iptables-alerter-9tbc5"
Sep 17 17:34:53.010297 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.009902    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f5a84a2-2e30-413a-823f-29cbf761cee0-ovn-node-metrics-cert\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.010297 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.009992    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-host-run-netns\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.010297 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.009995    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f5a84a2-2e30-413a-823f-29cbf761cee0-host-cni-netd\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.010297 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.010028    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ebc5fcd3-7a3c-4922-b880-c677bdb77777-cnibin\") pod \"multus-additional-cni-plugins-7rbtw\" (UID: \"ebc5fcd3-7a3c-4922-b880-c677bdb77777\") " pod="openshift-multus/multus-additional-cni-plugins-7rbtw"
Sep 17 17:34:53.010297 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:53.010065    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:53.010297 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.010119    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b92cf805-a323-422f-b021-6e8d9bce0099-tmp-dir\") pod \"node-resolver-gzt58\" (UID: \"b92cf805-a323-422f-b021-6e8d9bce0099\") " pod="openshift-dns/node-resolver-gzt58"
Sep 17 17:34:53.010986 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:53.010327    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae10190f-448c-4e69-a268-85f77c85d7b6-metrics-certs podName:ae10190f-448c-4e69-a268-85f77c85d7b6 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:53.510297595 +0000 UTC m=+4.601604345 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ae10190f-448c-4e69-a268-85f77c85d7b6-metrics-certs") pod "network-metrics-daemon-fh98p" (UID: "ae10190f-448c-4e69-a268-85f77c85d7b6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:53.010986 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.010457    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f5a84a2-2e30-413a-823f-29cbf761cee0-env-overrides\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.010986 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.010624    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3541d447-ae55-428c-9ab4-620f06f9c2fa-multus-daemon-config\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:53.014108 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.014036    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/798a6256-4107-44bb-9da2-b3a7c769eda1-agent-certs\") pod \"konnectivity-agent-gtn5h\" (UID: \"798a6256-4107-44bb-9da2-b3a7c769eda1\") " pod="kube-system/konnectivity-agent-gtn5h"
Sep 17 17:34:53.015013 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.014960    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gqg9\" (UniqueName: \"kubernetes.io/projected/ebc5fcd3-7a3c-4922-b880-c677bdb77777-kube-api-access-7gqg9\") pod \"multus-additional-cni-plugins-7rbtw\" (UID: \"ebc5fcd3-7a3c-4922-b880-c677bdb77777\") " pod="openshift-multus/multus-additional-cni-plugins-7rbtw"
Sep 17 17:34:53.015960 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.015937    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cflz\" (UniqueName: \"kubernetes.io/projected/3541d447-ae55-428c-9ab4-620f06f9c2fa-kube-api-access-2cflz\") pod \"multus-gnhzr\" (UID: \"3541d447-ae55-428c-9ab4-620f06f9c2fa\") " pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:53.017152 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.017132    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v45lp\" (UniqueName: \"kubernetes.io/projected/ae10190f-448c-4e69-a268-85f77c85d7b6-kube-api-access-v45lp\") pod \"network-metrics-daemon-fh98p\" (UID: \"ae10190f-448c-4e69-a268-85f77c85d7b6\") " pod="openshift-multus/network-metrics-daemon-fh98p"
Sep 17 17:34:53.017976 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.017953    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t4jw\" (UniqueName: \"kubernetes.io/projected/6f5a84a2-2e30-413a-823f-29cbf761cee0-kube-api-access-5t4jw\") pod \"ovnkube-node-9fhwv\" (UID: \"6f5a84a2-2e30-413a-823f-29cbf761cee0\") " pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.023642 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.023619    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9llwv\" (UniqueName: \"kubernetes.io/projected/57984009-dbb5-41b6-a1cb-d66608916495-kube-api-access-9llwv\") pod \"iptables-alerter-9tbc5\" (UID: \"57984009-dbb5-41b6-a1cb-d66608916495\") " pod="openshift-network-operator/iptables-alerter-9tbc5"
Sep 17 17:34:53.024576 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.024557    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsw9r\" (UniqueName: \"kubernetes.io/projected/a20d51c6-5fef-4251-9e8f-ee4318f97741-kube-api-access-xsw9r\") pod \"node-ca-ph456\" (UID: \"a20d51c6-5fef-4251-9e8f-ee4318f97741\") " pod="openshift-image-registry/node-ca-ph456"
Sep 17 17:34:53.024990 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.024973    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c226j\" (UniqueName: \"kubernetes.io/projected/b92cf805-a323-422f-b021-6e8d9bce0099-kube-api-access-c226j\") pod \"node-resolver-gzt58\" (UID: \"b92cf805-a323-422f-b021-6e8d9bce0099\") " pod="openshift-dns/node-resolver-gzt58"
Sep 17 17:34:53.038217 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.038176    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-30.ec2.internal" event={"ID":"5da666044a5091ad00b137b36d5ff04c","Type":"ContainerStarted","Data":"94e679826b76d487888e84846cafa11907aa47c0b7d8a686f4e16b276e20f596"}
Sep 17 17:34:53.039319 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.039289    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-30.ec2.internal" event={"ID":"710fcb8c6a77db50f998ce1b3ff8fde1","Type":"ContainerStarted","Data":"9e260962948a5ec73d99e0d5407940e4ff91d4bd21ae1aec6d45b1f7c4b7e560"}
Sep 17 17:34:53.110107 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110073    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-etc-systemd\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.110107 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110110    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c1593193-cc86-43b4-bfd5-75855a7d027d-socket-dir\") pod \"aws-ebs-csi-driver-node-9n586\" (UID: \"c1593193-cc86-43b4-bfd5-75855a7d027d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n586"
Sep 17 17:34:53.110325 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110137    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q862\" (UniqueName: \"kubernetes.io/projected/c1593193-cc86-43b4-bfd5-75855a7d027d-kube-api-access-8q862\") pod \"aws-ebs-csi-driver-node-9n586\" (UID: \"c1593193-cc86-43b4-bfd5-75855a7d027d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n586"
Sep 17 17:34:53.110325 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110162    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-etc-modprobe-d\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.110325 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110182    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-var-lib-kubelet\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.110325 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110205    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c1593193-cc86-43b4-bfd5-75855a7d027d-registration-dir\") pod \"aws-ebs-csi-driver-node-9n586\" (UID: \"c1593193-cc86-43b4-bfd5-75855a7d027d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n586"
Sep 17 17:34:53.110325 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110207    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-etc-systemd\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.110325 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110272    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c1593193-cc86-43b4-bfd5-75855a7d027d-registration-dir\") pod \"aws-ebs-csi-driver-node-9n586\" (UID: \"c1593193-cc86-43b4-bfd5-75855a7d027d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n586"
Sep 17 17:34:53.110325 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110305    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-var-lib-kubelet\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.110325 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110313    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-etc-modprobe-d\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.110650 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110326    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87775\" (UniqueName: \"kubernetes.io/projected/df33b3f5-aa93-4cb0-b1cb-3d89813c9bac-kube-api-access-87775\") pod \"network-check-target-ksb2q\" (UID: \"df33b3f5-aa93-4cb0-b1cb-3d89813c9bac\") " pod="openshift-network-diagnostics/network-check-target-ksb2q"
Sep 17 17:34:53.110650 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110348    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c1593193-cc86-43b4-bfd5-75855a7d027d-socket-dir\") pod \"aws-ebs-csi-driver-node-9n586\" (UID: \"c1593193-cc86-43b4-bfd5-75855a7d027d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n586"
Sep 17 17:34:53.110650 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110379    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-etc-sysctl-d\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.110650 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110405    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-sys\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.110650 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110443    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-tmp\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.110650 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110474    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c1593193-cc86-43b4-bfd5-75855a7d027d-sys-fs\") pod \"aws-ebs-csi-driver-node-9n586\" (UID: \"c1593193-cc86-43b4-bfd5-75855a7d027d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n586"
Sep 17 17:34:53.110650 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110493    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-etc-sysctl-d\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.110650 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110497    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-etc-tuned\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.110650 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110534    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-sys\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.110650 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110541    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c1593193-cc86-43b4-bfd5-75855a7d027d-device-dir\") pod \"aws-ebs-csi-driver-node-9n586\" (UID: \"c1593193-cc86-43b4-bfd5-75855a7d027d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n586"
Sep 17 17:34:53.110650 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110576    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-etc-sysctl-conf\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.110650 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110586    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c1593193-cc86-43b4-bfd5-75855a7d027d-device-dir\") pod \"aws-ebs-csi-driver-node-9n586\" (UID: \"c1593193-cc86-43b4-bfd5-75855a7d027d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n586"
Sep 17 17:34:53.110650 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110603    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-lib-modules\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.110650 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110646    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c1593193-cc86-43b4-bfd5-75855a7d027d-etc-selinux\") pod \"aws-ebs-csi-driver-node-9n586\" (UID: \"c1593193-cc86-43b4-bfd5-75855a7d027d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n586"
Sep 17 17:34:53.110650 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110648    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c1593193-cc86-43b4-bfd5-75855a7d027d-sys-fs\") pod \"aws-ebs-csi-driver-node-9n586\" (UID: \"c1593193-cc86-43b4-bfd5-75855a7d027d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n586"
Sep 17 17:34:53.111358 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110707    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-etc-sysctl-conf\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.111358 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110732    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-host\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.111358 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110753    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm2vq\" (UniqueName: \"kubernetes.io/projected/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-kube-api-access-vm2vq\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.111358 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110771    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c1593193-cc86-43b4-bfd5-75855a7d027d-etc-selinux\") pod \"aws-ebs-csi-driver-node-9n586\" (UID: \"c1593193-cc86-43b4-bfd5-75855a7d027d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n586"
Sep 17 17:34:53.111358 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110775    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-etc-sysconfig\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.111358 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110785    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-lib-modules\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.111358 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110791    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-host\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.111358 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110810    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-etc-sysconfig\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.111358 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110814    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-etc-kubernetes\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.111358 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110840    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-run\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.111358 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110866    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c1593193-cc86-43b4-bfd5-75855a7d027d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9n586\" (UID: \"c1593193-cc86-43b4-bfd5-75855a7d027d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n586"
Sep 17 17:34:53.111358 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110917    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-etc-kubernetes\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.111358 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110920    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-run\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.111358 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.110935    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c1593193-cc86-43b4-bfd5-75855a7d027d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9n586\" (UID: \"c1593193-cc86-43b4-bfd5-75855a7d027d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n586"
Sep 17 17:34:53.113097 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.113076    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-etc-tuned\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.113196 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.113112    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-tmp\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.117362 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:53.117315    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:53.117362 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:53.117343    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:53.117362 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:53.117355    2583 projected.go:196] Error preparing data for projected volume kube-api-access-87775 for pod openshift-network-diagnostics/network-check-target-ksb2q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:53.117550 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:53.117414    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df33b3f5-aa93-4cb0-b1cb-3d89813c9bac-kube-api-access-87775 podName:df33b3f5-aa93-4cb0-b1cb-3d89813c9bac nodeName:}" failed. No retries permitted until 2026-09-17 17:34:53.617395588 +0000 UTC m=+4.708702350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-87775" (UniqueName: "kubernetes.io/projected/df33b3f5-aa93-4cb0-b1cb-3d89813c9bac-kube-api-access-87775") pod "network-check-target-ksb2q" (UID: "df33b3f5-aa93-4cb0-b1cb-3d89813c9bac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:53.119850 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.119829    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q862\" (UniqueName: \"kubernetes.io/projected/c1593193-cc86-43b4-bfd5-75855a7d027d-kube-api-access-8q862\") pod \"aws-ebs-csi-driver-node-9n586\" (UID: \"c1593193-cc86-43b4-bfd5-75855a7d027d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n586"
Sep 17 17:34:53.120042 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.120021    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm2vq\" (UniqueName: \"kubernetes.io/projected/d4b4deeb-717d-412b-98e6-ea41d9f59ceb-kube-api-access-vm2vq\") pod \"tuned-h8287\" (UID: \"d4b4deeb-717d-412b-98e6-ea41d9f59ceb\") " pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.187799 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.187766    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gzt58"
Sep 17 17:34:53.197701 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.197678    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gnhzr"
Sep 17 17:34:53.205274 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.205252    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gtn5h"
Sep 17 17:34:53.216123 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.216104    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9tbc5"
Sep 17 17:34:53.222689 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.222671    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ph456"
Sep 17 17:34:53.229067 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.229051    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7rbtw"
Sep 17 17:34:53.235652 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.235628    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:34:53.242330 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.242310    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n586"
Sep 17 17:34:53.247976 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.247959    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h8287"
Sep 17 17:34:53.350449 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.350422    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:53.515241 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.515176    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae10190f-448c-4e69-a268-85f77c85d7b6-metrics-certs\") pod \"network-metrics-daemon-fh98p\" (UID: \"ae10190f-448c-4e69-a268-85f77c85d7b6\") " pod="openshift-multus/network-metrics-daemon-fh98p"
Sep 17 17:34:53.515241 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.515220    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4696b495-138f-4b5a-838b-d13ec3d8a3a6-original-pull-secret\") pod \"global-pull-secret-syncer-c46w4\" (UID: \"4696b495-138f-4b5a-838b-d13ec3d8a3a6\") " pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:34:53.515416 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:53.515330    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:53.515416 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:53.515394    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae10190f-448c-4e69-a268-85f77c85d7b6-metrics-certs podName:ae10190f-448c-4e69-a268-85f77c85d7b6 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:54.515379736 +0000 UTC m=+5.606686479 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ae10190f-448c-4e69-a268-85f77c85d7b6-metrics-certs") pod "network-metrics-daemon-fh98p" (UID: "ae10190f-448c-4e69-a268-85f77c85d7b6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:53.515486 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:53.515330    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:53.515486 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:53.515466    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4696b495-138f-4b5a-838b-d13ec3d8a3a6-original-pull-secret podName:4696b495-138f-4b5a-838b-d13ec3d8a3a6 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:54.515454838 +0000 UTC m=+5.606761581 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4696b495-138f-4b5a-838b-d13ec3d8a3a6-original-pull-secret") pod "global-pull-secret-syncer-c46w4" (UID: "4696b495-138f-4b5a-838b-d13ec3d8a3a6") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:53.716879 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.716834    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87775\" (UniqueName: \"kubernetes.io/projected/df33b3f5-aa93-4cb0-b1cb-3d89813c9bac-kube-api-access-87775\") pod \"network-check-target-ksb2q\" (UID: \"df33b3f5-aa93-4cb0-b1cb-3d89813c9bac\") " pod="openshift-network-diagnostics/network-check-target-ksb2q"
Sep 17 17:34:53.717062 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:53.716986    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:53.717062 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:53.717002    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:53.717062 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:53.717011    2583 projected.go:196] Error preparing data for projected volume kube-api-access-87775 for pod openshift-network-diagnostics/network-check-target-ksb2q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:53.717062 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:53.717055    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df33b3f5-aa93-4cb0-b1cb-3d89813c9bac-kube-api-access-87775 podName:df33b3f5-aa93-4cb0-b1cb-3d89813c9bac nodeName:}" failed. No retries permitted until 2026-09-17 17:34:54.71704343 +0000 UTC m=+5.808350187 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-87775" (UniqueName: "kubernetes.io/projected/df33b3f5-aa93-4cb0-b1cb-3d89813c9bac-kube-api-access-87775") pod "network-check-target-ksb2q" (UID: "df33b3f5-aa93-4cb0-b1cb-3d89813c9bac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:53.861294 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:53.861264    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f5a84a2_2e30_413a_823f_29cbf761cee0.slice/crio-340b133942bb5caf7acd060393a1e184816442d8f0162f629fc077c5a4456d50 WatchSource:0}: Error finding container 340b133942bb5caf7acd060393a1e184816442d8f0162f629fc077c5a4456d50: Status 404 returned error can't find the container with id 340b133942bb5caf7acd060393a1e184816442d8f0162f629fc077c5a4456d50
Sep 17 17:34:53.862275 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:53.862252    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b4deeb_717d_412b_98e6_ea41d9f59ceb.slice/crio-6ff85c5897a8f1ff0bc0ff1dca917cc4ce822b4ad44aee4fc9089bb75b9f0017 WatchSource:0}: Error finding container 6ff85c5897a8f1ff0bc0ff1dca917cc4ce822b4ad44aee4fc9089bb75b9f0017: Status 404 returned error can't find the container with id 6ff85c5897a8f1ff0bc0ff1dca917cc4ce822b4ad44aee4fc9089bb75b9f0017
Sep 17 17:34:53.863079 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:53.863050    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1593193_cc86_43b4_bfd5_75855a7d027d.slice/crio-d7d41a7255605809abe54c73dfa288190944c65d173479c3e1715c08d5c80de3 WatchSource:0}: Error finding container d7d41a7255605809abe54c73dfa288190944c65d173479c3e1715c08d5c80de3: Status 404 returned error can't find the container with id d7d41a7255605809abe54c73dfa288190944c65d173479c3e1715c08d5c80de3
Sep 17 17:34:53.863786 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:53.863646    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb92cf805_a323_422f_b021_6e8d9bce0099.slice/crio-c8adfdc52c18c2ac94656b5075ada2d80295c53973aadae164676558544f8ad0 WatchSource:0}: Error finding container c8adfdc52c18c2ac94656b5075ada2d80295c53973aadae164676558544f8ad0: Status 404 returned error can't find the container with id c8adfdc52c18c2ac94656b5075ada2d80295c53973aadae164676558544f8ad0
Sep 17 17:34:53.885695 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:53.885643    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda20d51c6_5fef_4251_9e8f_ee4318f97741.slice/crio-839d2e6fe6f823f2d4d2441f107b95a1325099c932f7a2c57aef7abc18acf5e3 WatchSource:0}: Error finding container 839d2e6fe6f823f2d4d2441f107b95a1325099c932f7a2c57aef7abc18acf5e3: Status 404 returned error can't find the container with id 839d2e6fe6f823f2d4d2441f107b95a1325099c932f7a2c57aef7abc18acf5e3
Sep 17 17:34:53.888334 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:53.888310    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod798a6256_4107_44bb_9da2_b3a7c769eda1.slice/crio-0330aac4d7bd3906555a1700507a2385ea3fe4723f06fa429ced8c37a2856686 WatchSource:0}: Error finding container 0330aac4d7bd3906555a1700507a2385ea3fe4723f06fa429ced8c37a2856686: Status 404 returned error can't find the container with id 0330aac4d7bd3906555a1700507a2385ea3fe4723f06fa429ced8c37a2856686
Sep 17 17:34:53.889084 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:53.889064    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3541d447_ae55_428c_9ab4_620f06f9c2fa.slice/crio-e01b16b4dcd3dab01be19ab272a9a3bd43d101e7fc2467c2f6c06871c9177b9e WatchSource:0}: Error finding container e01b16b4dcd3dab01be19ab272a9a3bd43d101e7fc2467c2f6c06871c9177b9e: Status 404 returned error can't find the container with id e01b16b4dcd3dab01be19ab272a9a3bd43d101e7fc2467c2f6c06871c9177b9e
Sep 17 17:34:53.890641 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:53.890373    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc5fcd3_7a3c_4922_b880_c677bdb77777.slice/crio-50857c96061c0a777422003cdd4b5f7ac7bf76551e8a423945e1dc7e96166856 WatchSource:0}: Error finding container 50857c96061c0a777422003cdd4b5f7ac7bf76551e8a423945e1dc7e96166856: Status 404 returned error can't find the container with id 50857c96061c0a777422003cdd4b5f7ac7bf76551e8a423945e1dc7e96166856
Sep 17 17:34:53.893016 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:34:53.892432    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57984009_dbb5_41b6_a1cb_d66608916495.slice/crio-39979a5bfe42529673db3dee95a3c32ec3b5661579d1269407335ed34fd927d7 WatchSource:0}: Error finding container 39979a5bfe42529673db3dee95a3c32ec3b5661579d1269407335ed34fd927d7: Status 404 returned error can't find the container with id 39979a5bfe42529673db3dee95a3c32ec3b5661579d1269407335ed34fd927d7
Sep 17 17:34:53.967725 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.967672    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:29:52 +0000 UTC" deadline="2028-04-12 15:58:56.398422052 +0000 UTC"
Sep 17 17:34:53.967725 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:53.967722    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13750h24m2.430703127s"
Sep 17 17:34:54.044363 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:54.044327    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-30.ec2.internal" event={"ID":"710fcb8c6a77db50f998ce1b3ff8fde1","Type":"ContainerStarted","Data":"dfb9a2e6833e14477f99c7b82111f364eb9fc922c711f2973dcdf713f1831187"}
Sep 17 17:34:54.045541 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:54.045516    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ph456" event={"ID":"a20d51c6-5fef-4251-9e8f-ee4318f97741","Type":"ContainerStarted","Data":"839d2e6fe6f823f2d4d2441f107b95a1325099c932f7a2c57aef7abc18acf5e3"}
Sep 17 17:34:54.046627 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:54.046599    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gzt58" event={"ID":"b92cf805-a323-422f-b021-6e8d9bce0099","Type":"ContainerStarted","Data":"c8adfdc52c18c2ac94656b5075ada2d80295c53973aadae164676558544f8ad0"}
Sep 17 17:34:54.048450 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:54.048423    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h8287" event={"ID":"d4b4deeb-717d-412b-98e6-ea41d9f59ceb","Type":"ContainerStarted","Data":"6ff85c5897a8f1ff0bc0ff1dca917cc4ce822b4ad44aee4fc9089bb75b9f0017"}
Sep 17 17:34:54.050763 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:54.050738    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv" event={"ID":"6f5a84a2-2e30-413a-823f-29cbf761cee0","Type":"ContainerStarted","Data":"340b133942bb5caf7acd060393a1e184816442d8f0162f629fc077c5a4456d50"}
Sep 17 17:34:54.051931 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:54.051910    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9tbc5" event={"ID":"57984009-dbb5-41b6-a1cb-d66608916495","Type":"ContainerStarted","Data":"39979a5bfe42529673db3dee95a3c32ec3b5661579d1269407335ed34fd927d7"}
Sep 17 17:34:54.053498 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:54.053478    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rbtw" event={"ID":"ebc5fcd3-7a3c-4922-b880-c677bdb77777","Type":"ContainerStarted","Data":"50857c96061c0a777422003cdd4b5f7ac7bf76551e8a423945e1dc7e96166856"}
Sep 17 17:34:54.055108 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:54.055088    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gnhzr" event={"ID":"3541d447-ae55-428c-9ab4-620f06f9c2fa","Type":"ContainerStarted","Data":"e01b16b4dcd3dab01be19ab272a9a3bd43d101e7fc2467c2f6c06871c9177b9e"}
Sep 17 17:34:54.056148 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:54.056126    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gtn5h" event={"ID":"798a6256-4107-44bb-9da2-b3a7c769eda1","Type":"ContainerStarted","Data":"0330aac4d7bd3906555a1700507a2385ea3fe4723f06fa429ced8c37a2856686"}
Sep 17 17:34:54.057276 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:54.057254    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n586" event={"ID":"c1593193-cc86-43b4-bfd5-75855a7d027d","Type":"ContainerStarted","Data":"d7d41a7255605809abe54c73dfa288190944c65d173479c3e1715c08d5c80de3"}
Sep 17 17:34:54.058061 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:54.058014    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-30.ec2.internal" podStartSLOduration=3.057997584 podStartE2EDuration="3.057997584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:54.057229931 +0000 UTC m=+5.148536721" watchObservedRunningTime="2026-09-17 17:34:54.057997584 +0000 UTC m=+5.149304349"
Sep 17 17:34:54.524120 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:54.522976    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae10190f-448c-4e69-a268-85f77c85d7b6-metrics-certs\") pod \"network-metrics-daemon-fh98p\" (UID: \"ae10190f-448c-4e69-a268-85f77c85d7b6\") " pod="openshift-multus/network-metrics-daemon-fh98p"
Sep 17 17:34:54.524120 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:54.523247    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4696b495-138f-4b5a-838b-d13ec3d8a3a6-original-pull-secret\") pod \"global-pull-secret-syncer-c46w4\" (UID: \"4696b495-138f-4b5a-838b-d13ec3d8a3a6\") " pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:34:54.524120 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:54.523383    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:54.524120 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:54.523445    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4696b495-138f-4b5a-838b-d13ec3d8a3a6-original-pull-secret podName:4696b495-138f-4b5a-838b-d13ec3d8a3a6 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:56.5234268 +0000 UTC m=+7.614733556 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4696b495-138f-4b5a-838b-d13ec3d8a3a6-original-pull-secret") pod "global-pull-secret-syncer-c46w4" (UID: "4696b495-138f-4b5a-838b-d13ec3d8a3a6") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:54.524120 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:54.523940    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:54.524120 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:54.523994    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae10190f-448c-4e69-a268-85f77c85d7b6-metrics-certs podName:ae10190f-448c-4e69-a268-85f77c85d7b6 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:56.523975825 +0000 UTC m=+7.615282585 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ae10190f-448c-4e69-a268-85f77c85d7b6-metrics-certs") pod "network-metrics-daemon-fh98p" (UID: "ae10190f-448c-4e69-a268-85f77c85d7b6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:54.725426 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:54.725391    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87775\" (UniqueName: \"kubernetes.io/projected/df33b3f5-aa93-4cb0-b1cb-3d89813c9bac-kube-api-access-87775\") pod \"network-check-target-ksb2q\" (UID: \"df33b3f5-aa93-4cb0-b1cb-3d89813c9bac\") " pod="openshift-network-diagnostics/network-check-target-ksb2q"
Sep 17 17:34:54.725610 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:54.725560    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:54.725610 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:54.725578    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:54.725610 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:54.725590    2583 projected.go:196] Error preparing data for projected volume kube-api-access-87775 for pod openshift-network-diagnostics/network-check-target-ksb2q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:54.725852 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:54.725685    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df33b3f5-aa93-4cb0-b1cb-3d89813c9bac-kube-api-access-87775 podName:df33b3f5-aa93-4cb0-b1cb-3d89813c9bac nodeName:}" failed. No retries permitted until 2026-09-17 17:34:56.725651666 +0000 UTC m=+7.816958415 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-87775" (UniqueName: "kubernetes.io/projected/df33b3f5-aa93-4cb0-b1cb-3d89813c9bac-kube-api-access-87775") pod "network-check-target-ksb2q" (UID: "df33b3f5-aa93-4cb0-b1cb-3d89813c9bac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:54.968409 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:54.968317    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:29:52 +0000 UTC" deadline="2028-03-15 06:01:54.1876282 +0000 UTC"
Sep 17 17:34:54.968409 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:54.968358    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13068h26m59.219274666s"
Sep 17 17:34:55.036516 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:55.035776    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh98p"
Sep 17 17:34:55.036516 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:55.035911    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh98p" podUID="ae10190f-448c-4e69-a268-85f77c85d7b6"
Sep 17 17:34:55.036516 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:55.036283    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksb2q"
Sep 17 17:34:55.036516 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:55.036368    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksb2q" podUID="df33b3f5-aa93-4cb0-b1cb-3d89813c9bac"
Sep 17 17:34:55.037441 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:55.037071    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:34:55.037441 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:55.037168    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c46w4" podUID="4696b495-138f-4b5a-838b-d13ec3d8a3a6"
Sep 17 17:34:55.078683 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:55.078166    2583 generic.go:356] "Generic (PLEG): container finished" podID="5da666044a5091ad00b137b36d5ff04c" containerID="54a0a6410384b1a110e9173f0ee50f2761b43a34b6f881c4f8ef8020aee38128" exitCode=0
Sep 17 17:34:55.078904 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:55.078851    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-30.ec2.internal" event={"ID":"5da666044a5091ad00b137b36d5ff04c","Type":"ContainerDied","Data":"54a0a6410384b1a110e9173f0ee50f2761b43a34b6f881c4f8ef8020aee38128"}
Sep 17 17:34:56.110018 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:56.109164    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-30.ec2.internal" event={"ID":"5da666044a5091ad00b137b36d5ff04c","Type":"ContainerStarted","Data":"cac24eb1659ff2bbd95cbae48fef0eaa46f47f06c7f6d522f1e3a5d6cb38e97e"}
Sep 17 17:34:56.541476 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:56.541391    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae10190f-448c-4e69-a268-85f77c85d7b6-metrics-certs\") pod \"network-metrics-daemon-fh98p\" (UID: \"ae10190f-448c-4e69-a268-85f77c85d7b6\") " pod="openshift-multus/network-metrics-daemon-fh98p"
Sep 17 17:34:56.541476 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:56.541466    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4696b495-138f-4b5a-838b-d13ec3d8a3a6-original-pull-secret\") pod \"global-pull-secret-syncer-c46w4\" (UID: \"4696b495-138f-4b5a-838b-d13ec3d8a3a6\") " pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:34:56.541702 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:56.541605    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:56.541702 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:56.541680    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4696b495-138f-4b5a-838b-d13ec3d8a3a6-original-pull-secret podName:4696b495-138f-4b5a-838b-d13ec3d8a3a6 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:00.541646576 +0000 UTC m=+11.632953324 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4696b495-138f-4b5a-838b-d13ec3d8a3a6-original-pull-secret") pod "global-pull-secret-syncer-c46w4" (UID: "4696b495-138f-4b5a-838b-d13ec3d8a3a6") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:56.542098 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:56.542077    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:56.542189 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:56.542134    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae10190f-448c-4e69-a268-85f77c85d7b6-metrics-certs podName:ae10190f-448c-4e69-a268-85f77c85d7b6 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:00.54211861 +0000 UTC m=+11.633425356 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ae10190f-448c-4e69-a268-85f77c85d7b6-metrics-certs") pod "network-metrics-daemon-fh98p" (UID: "ae10190f-448c-4e69-a268-85f77c85d7b6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:56.743253 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:56.743215    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87775\" (UniqueName: \"kubernetes.io/projected/df33b3f5-aa93-4cb0-b1cb-3d89813c9bac-kube-api-access-87775\") pod \"network-check-target-ksb2q\" (UID: \"df33b3f5-aa93-4cb0-b1cb-3d89813c9bac\") " pod="openshift-network-diagnostics/network-check-target-ksb2q"
Sep 17 17:34:56.743877 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:56.743420    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:56.743877 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:56.743453    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:56.743877 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:56.743468    2583 projected.go:196] Error preparing data for projected volume kube-api-access-87775 for pod openshift-network-diagnostics/network-check-target-ksb2q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:56.743877 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:56.743533    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df33b3f5-aa93-4cb0-b1cb-3d89813c9bac-kube-api-access-87775 podName:df33b3f5-aa93-4cb0-b1cb-3d89813c9bac nodeName:}" failed. No retries permitted until 2026-09-17 17:35:00.743514839 +0000 UTC m=+11.834821596 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-87775" (UniqueName: "kubernetes.io/projected/df33b3f5-aa93-4cb0-b1cb-3d89813c9bac-kube-api-access-87775") pod "network-check-target-ksb2q" (UID: "df33b3f5-aa93-4cb0-b1cb-3d89813c9bac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:57.035840 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:57.035766    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh98p"
Sep 17 17:34:57.035840 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:57.035806    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:34:57.036056 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:57.035904    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh98p" podUID="ae10190f-448c-4e69-a268-85f77c85d7b6"
Sep 17 17:34:57.036521 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:57.036294    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c46w4" podUID="4696b495-138f-4b5a-838b-d13ec3d8a3a6"
Sep 17 17:34:57.036521 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:57.036373    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksb2q"
Sep 17 17:34:57.036521 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:57.036443    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksb2q" podUID="df33b3f5-aa93-4cb0-b1cb-3d89813c9bac"
Sep 17 17:34:59.037317 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:59.037281    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksb2q"
Sep 17 17:34:59.037317 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:59.037305    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh98p"
Sep 17 17:34:59.037879 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:59.037413    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksb2q" podUID="df33b3f5-aa93-4cb0-b1cb-3d89813c9bac"
Sep 17 17:34:59.037879 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:34:59.037465    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:34:59.037879 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:59.037634    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh98p" podUID="ae10190f-448c-4e69-a268-85f77c85d7b6"
Sep 17 17:34:59.037879 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:34:59.037742    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c46w4" podUID="4696b495-138f-4b5a-838b-d13ec3d8a3a6"
Sep 17 17:35:00.574975 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:00.574927    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4696b495-138f-4b5a-838b-d13ec3d8a3a6-original-pull-secret\") pod \"global-pull-secret-syncer-c46w4\" (UID: \"4696b495-138f-4b5a-838b-d13ec3d8a3a6\") " pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:35:00.575385 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:00.575013    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae10190f-448c-4e69-a268-85f77c85d7b6-metrics-certs\") pod \"network-metrics-daemon-fh98p\" (UID: \"ae10190f-448c-4e69-a268-85f77c85d7b6\") " pod="openshift-multus/network-metrics-daemon-fh98p"
Sep 17 17:35:00.575385 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:00.575166    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:00.575385 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:00.575234    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae10190f-448c-4e69-a268-85f77c85d7b6-metrics-certs podName:ae10190f-448c-4e69-a268-85f77c85d7b6 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:08.575211151 +0000 UTC m=+19.666517915 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ae10190f-448c-4e69-a268-85f77c85d7b6-metrics-certs") pod "network-metrics-daemon-fh98p" (UID: "ae10190f-448c-4e69-a268-85f77c85d7b6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:00.575598 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:00.575537    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:00.575598 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:00.575576    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4696b495-138f-4b5a-838b-d13ec3d8a3a6-original-pull-secret podName:4696b495-138f-4b5a-838b-d13ec3d8a3a6 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:08.575567155 +0000 UTC m=+19.666873899 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4696b495-138f-4b5a-838b-d13ec3d8a3a6-original-pull-secret") pod "global-pull-secret-syncer-c46w4" (UID: "4696b495-138f-4b5a-838b-d13ec3d8a3a6") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:00.777355 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:00.777317    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87775\" (UniqueName: \"kubernetes.io/projected/df33b3f5-aa93-4cb0-b1cb-3d89813c9bac-kube-api-access-87775\") pod \"network-check-target-ksb2q\" (UID: \"df33b3f5-aa93-4cb0-b1cb-3d89813c9bac\") " pod="openshift-network-diagnostics/network-check-target-ksb2q"
Sep 17 17:35:00.777562 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:00.777541    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:35:00.777670 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:00.777574    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:35:00.777670 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:00.777589    2583 projected.go:196] Error preparing data for projected volume kube-api-access-87775 for pod openshift-network-diagnostics/network-check-target-ksb2q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:00.777670 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:00.777654    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df33b3f5-aa93-4cb0-b1cb-3d89813c9bac-kube-api-access-87775 podName:df33b3f5-aa93-4cb0-b1cb-3d89813c9bac nodeName:}" failed. No retries permitted until 2026-09-17 17:35:08.777635734 +0000 UTC m=+19.868942484 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-87775" (UniqueName: "kubernetes.io/projected/df33b3f5-aa93-4cb0-b1cb-3d89813c9bac-kube-api-access-87775") pod "network-check-target-ksb2q" (UID: "df33b3f5-aa93-4cb0-b1cb-3d89813c9bac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:01.035983 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:01.035680    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksb2q"
Sep 17 17:35:01.035983 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:01.035848    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksb2q" podUID="df33b3f5-aa93-4cb0-b1cb-3d89813c9bac"
Sep 17 17:35:01.036601 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:01.036227    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:35:01.036601 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:01.036327    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c46w4" podUID="4696b495-138f-4b5a-838b-d13ec3d8a3a6"
Sep 17 17:35:01.036601 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:01.036443    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh98p"
Sep 17 17:35:01.036601 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:01.036558    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh98p" podUID="ae10190f-448c-4e69-a268-85f77c85d7b6"
Sep 17 17:35:03.034782 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:03.034746    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksb2q"
Sep 17 17:35:03.035237 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:03.034748    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:35:03.035237 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:03.034892    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksb2q" podUID="df33b3f5-aa93-4cb0-b1cb-3d89813c9bac"
Sep 17 17:35:03.035237 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:03.034747    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh98p"
Sep 17 17:35:03.035237 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:03.034954    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c46w4" podUID="4696b495-138f-4b5a-838b-d13ec3d8a3a6"
Sep 17 17:35:03.035237 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:03.035027    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh98p" podUID="ae10190f-448c-4e69-a268-85f77c85d7b6"
Sep 17 17:35:05.035081 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:05.035029    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh98p"
Sep 17 17:35:05.035565 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:05.035039    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksb2q"
Sep 17 17:35:05.035565 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:05.035167    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:35:05.035565 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:05.035166    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh98p" podUID="ae10190f-448c-4e69-a268-85f77c85d7b6"
Sep 17 17:35:05.035565 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:05.035275    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksb2q" podUID="df33b3f5-aa93-4cb0-b1cb-3d89813c9bac"
Sep 17 17:35:05.035565 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:05.035378    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c46w4" podUID="4696b495-138f-4b5a-838b-d13ec3d8a3a6"
Sep 17 17:35:07.035144 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:07.035108    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh98p"
Sep 17 17:35:07.035639 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:07.035114    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksb2q"
Sep 17 17:35:07.035639 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:07.035269    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:35:07.035639 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:07.035349    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksb2q" podUID="df33b3f5-aa93-4cb0-b1cb-3d89813c9bac"
Sep 17 17:35:07.035639 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:07.035236    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh98p" podUID="ae10190f-448c-4e69-a268-85f77c85d7b6"
Sep 17 17:35:07.035639 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:07.035449    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c46w4" podUID="4696b495-138f-4b5a-838b-d13ec3d8a3a6"
Sep 17 17:35:08.630732 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:08.630688    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4696b495-138f-4b5a-838b-d13ec3d8a3a6-original-pull-secret\") pod \"global-pull-secret-syncer-c46w4\" (UID: \"4696b495-138f-4b5a-838b-d13ec3d8a3a6\") " pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:35:08.631233 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:08.630872    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae10190f-448c-4e69-a268-85f77c85d7b6-metrics-certs\") pod \"network-metrics-daemon-fh98p\" (UID: \"ae10190f-448c-4e69-a268-85f77c85d7b6\") " pod="openshift-multus/network-metrics-daemon-fh98p"
Sep 17 17:35:08.631233 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:08.630883    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:08.631233 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:08.630983    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4696b495-138f-4b5a-838b-d13ec3d8a3a6-original-pull-secret podName:4696b495-138f-4b5a-838b-d13ec3d8a3a6 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:24.630966648 +0000 UTC m=+35.722273397 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4696b495-138f-4b5a-838b-d13ec3d8a3a6-original-pull-secret") pod "global-pull-secret-syncer-c46w4" (UID: "4696b495-138f-4b5a-838b-d13ec3d8a3a6") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:08.631233 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:08.631001    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:08.631233 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:08.631053    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae10190f-448c-4e69-a268-85f77c85d7b6-metrics-certs podName:ae10190f-448c-4e69-a268-85f77c85d7b6 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:24.631039911 +0000 UTC m=+35.722346682 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ae10190f-448c-4e69-a268-85f77c85d7b6-metrics-certs") pod "network-metrics-daemon-fh98p" (UID: "ae10190f-448c-4e69-a268-85f77c85d7b6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:08.832762 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:08.832718    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87775\" (UniqueName: \"kubernetes.io/projected/df33b3f5-aa93-4cb0-b1cb-3d89813c9bac-kube-api-access-87775\") pod \"network-check-target-ksb2q\" (UID: \"df33b3f5-aa93-4cb0-b1cb-3d89813c9bac\") " pod="openshift-network-diagnostics/network-check-target-ksb2q"
Sep 17 17:35:08.832935 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:08.832917    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:35:08.832983 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:08.832944    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:35:08.832983 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:08.832959    2583 projected.go:196] Error preparing data for projected volume kube-api-access-87775 for pod openshift-network-diagnostics/network-check-target-ksb2q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:08.833068 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:08.833028    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df33b3f5-aa93-4cb0-b1cb-3d89813c9bac-kube-api-access-87775 podName:df33b3f5-aa93-4cb0-b1cb-3d89813c9bac nodeName:}" failed. No retries permitted until 2026-09-17 17:35:24.833008701 +0000 UTC m=+35.924315464 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-87775" (UniqueName: "kubernetes.io/projected/df33b3f5-aa93-4cb0-b1cb-3d89813c9bac-kube-api-access-87775") pod "network-check-target-ksb2q" (UID: "df33b3f5-aa93-4cb0-b1cb-3d89813c9bac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:09.034716 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:09.034626    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh98p"
Sep 17 17:35:09.034870 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:09.034626    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksb2q"
Sep 17 17:35:09.034870 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:09.034788    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:35:09.035418 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:09.035318    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksb2q" podUID="df33b3f5-aa93-4cb0-b1cb-3d89813c9bac"
Sep 17 17:35:09.035603 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:09.035572    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c46w4" podUID="4696b495-138f-4b5a-838b-d13ec3d8a3a6"
Sep 17 17:35:09.036504 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:09.034785    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh98p" podUID="ae10190f-448c-4e69-a268-85f77c85d7b6"
Sep 17 17:35:11.035494 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:11.035461    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:35:11.035945 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:11.035598    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c46w4" podUID="4696b495-138f-4b5a-838b-d13ec3d8a3a6"
Sep 17 17:35:11.035945 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:11.035728    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh98p"
Sep 17 17:35:11.035945 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:11.035848    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh98p" podUID="ae10190f-448c-4e69-a268-85f77c85d7b6"
Sep 17 17:35:11.035945 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:11.035885    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksb2q"
Sep 17 17:35:11.036080 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:11.035949    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksb2q" podUID="df33b3f5-aa93-4cb0-b1cb-3d89813c9bac"
Sep 17 17:35:12.142449 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:12.142234    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv" event={"ID":"6f5a84a2-2e30-413a-823f-29cbf761cee0","Type":"ContainerStarted","Data":"986b3306b1b4b5e393c723c6b678835396f241d0d5bbf38de3846d2352d0baa4"}
Sep 17 17:35:12.144050 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:12.144022    2583 generic.go:356] "Generic (PLEG): container finished" podID="ebc5fcd3-7a3c-4922-b880-c677bdb77777" containerID="5b108840cd667dcdb74a154ee18e5c1f86ac8a9ed5090800a698d30dd9e18289" exitCode=0
Sep 17 17:35:12.144153 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:12.144101    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rbtw" event={"ID":"ebc5fcd3-7a3c-4922-b880-c677bdb77777","Type":"ContainerDied","Data":"5b108840cd667dcdb74a154ee18e5c1f86ac8a9ed5090800a698d30dd9e18289"}
Sep 17 17:35:12.146156 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:12.146130    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gnhzr" event={"ID":"3541d447-ae55-428c-9ab4-620f06f9c2fa","Type":"ContainerStarted","Data":"7fd684b6ace74a1d928482bbf1479eb7e860e590f2afaa577cc3c71ae083f453"}
Sep 17 17:35:12.147758 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:12.147737    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gtn5h" event={"ID":"798a6256-4107-44bb-9da2-b3a7c769eda1","Type":"ContainerStarted","Data":"86b57935d1ad01d9375106eb0f8ed3c1c3df82ec6df032d7f5471bc6450eca1f"}
Sep 17 17:35:12.150955 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:12.150297    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n586" event={"ID":"c1593193-cc86-43b4-bfd5-75855a7d027d","Type":"ContainerStarted","Data":"de77f2f782d086197940198c912779a3f709d747a10f0396a3ca3c4c13e47abc"}
Sep 17 17:35:12.152650 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:12.152627    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ph456" event={"ID":"a20d51c6-5fef-4251-9e8f-ee4318f97741","Type":"ContainerStarted","Data":"b4907e84c259c7a7ddc614ee98e88713eb5eee72da1e60a1f73d5f43e40b6a9a"}
Sep 17 17:35:12.154124 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:12.154090    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gzt58" event={"ID":"b92cf805-a323-422f-b021-6e8d9bce0099","Type":"ContainerStarted","Data":"c4e295cd718ac070eaf9a3698cbc125ae78539683c9b3147dda0bd08eae13499"}
Sep 17 17:35:12.155631 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:12.155613    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h8287" event={"ID":"d4b4deeb-717d-412b-98e6-ea41d9f59ceb","Type":"ContainerStarted","Data":"03fb07f2d82cfa88bcad9a71f854575a6b8dd49b60b03d01775b288354677f6b"}
Sep 17 17:35:12.170222 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:12.170181    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-30.ec2.internal" podStartSLOduration=20.170169041 podStartE2EDuration="20.170169041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:56.125831345 +0000 UTC m=+7.217138127" watchObservedRunningTime="2026-09-17 17:35:12.170169041 +0000 UTC m=+23.261475807"
Sep 17 17:35:12.188574 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:12.188527    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gnhzr" podStartSLOduration=3.6919788159999998 podStartE2EDuration="21.188509313s" totalImagesPullingTime="17.496530497s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:12.188335202 +0000 UTC m=+23.279641988" watchObservedRunningTime="2026-09-17 17:35:12.188509313 +0000 UTC m=+23.279816080"
Sep 17 17:35:12.208486 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:12.208439    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-h8287" podStartSLOduration=3.747409931 podStartE2EDuration="21.20842311s" totalImagesPullingTime="17.461013179s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:12.20769273 +0000 UTC m=+23.298999513" watchObservedRunningTime="2026-09-17 17:35:12.20842311 +0000 UTC m=+23.299729875"
Sep 17 17:35:12.232255 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:12.232207    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gzt58" podStartSLOduration=3.74577022 podStartE2EDuration="21.23218836s" totalImagesPullingTime="17.48641814s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:12.231854269 +0000 UTC m=+23.323161035" watchObservedRunningTime="2026-09-17 17:35:12.23218836 +0000 UTC m=+23.323495126"
Sep 17 17:35:12.257941 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:12.257904    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gtn5h" podStartSLOduration=3.852414385 podStartE2EDuration="21.257890847s" totalImagesPullingTime="17.405476462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:12.257847245 +0000 UTC m=+23.349154031" watchObservedRunningTime="2026-09-17 17:35:12.257890847 +0000 UTC m=+23.349197613"
Sep 17 17:35:12.280565 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:12.280508    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ph456" podStartSLOduration=3.803712875 podStartE2EDuration="21.280492104s" totalImagesPullingTime="17.476779229s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:12.280081531 +0000 UTC m=+23.371388325" watchObservedRunningTime="2026-09-17 17:35:12.280492104 +0000 UTC m=+23.371798869"
Sep 17 17:35:12.334583 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:12.334557    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gtn5h"
Sep 17 17:35:12.335261 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:12.335242    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gtn5h"
Sep 17 17:35:12.891700 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:12.891487    2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 17:35:12.985964 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:12.985809    2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T17:35:12.891697051Z","UUID":"f1e6a4f4-e7d9-4588-adbd-d1e2c8b105dc","Handler":null,"Name":"","Endpoint":""}
Sep 17 17:35:12.987572 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:12.987549    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 17:35:12.987698 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:12.987588    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 17:35:13.034928 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:13.034898    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:35:13.035097 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:13.034898    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksb2q"
Sep 17 17:35:13.035097 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:13.035025    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c46w4" podUID="4696b495-138f-4b5a-838b-d13ec3d8a3a6"
Sep 17 17:35:13.035097 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:13.035077    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksb2q" podUID="df33b3f5-aa93-4cb0-b1cb-3d89813c9bac"
Sep 17 17:35:13.035262 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:13.034899    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh98p"
Sep 17 17:35:13.035262 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:13.035217    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh98p" podUID="ae10190f-448c-4e69-a268-85f77c85d7b6"
Sep 17 17:35:13.160090 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:13.160054    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n586" event={"ID":"c1593193-cc86-43b4-bfd5-75855a7d027d","Type":"ContainerStarted","Data":"b3c1e430d382cd6beb6cbf28476bba868670a0a830a30f31b6d917faad603904"}
Sep 17 17:35:13.163458 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:13.163433    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv" event={"ID":"6f5a84a2-2e30-413a-823f-29cbf761cee0","Type":"ContainerStarted","Data":"8fc4e4dcb8ad4c50e16ce99a11060c4c124e5ac51db8c96268757c1ca0be82e2"}
Sep 17 17:35:13.163589 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:13.163513    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv" event={"ID":"6f5a84a2-2e30-413a-823f-29cbf761cee0","Type":"ContainerStarted","Data":"5cfe90f25e51a9ea0a95ae6acf2a02b504f573ec9e0d66e93028938ab17ed0f1"}
Sep 17 17:35:13.163589 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:13.163564    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv" event={"ID":"6f5a84a2-2e30-413a-823f-29cbf761cee0","Type":"ContainerStarted","Data":"2d33b7faff4432d730199b2bf6ec2da2a4ba08c52866f1871761597731f6cb98"}
Sep 17 17:35:13.163709 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:13.163610    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv" event={"ID":"6f5a84a2-2e30-413a-823f-29cbf761cee0","Type":"ContainerStarted","Data":"fda267007ab3501fc547f7de5dfa6768eac5b8ded82bb8db31654462a374c3e1"}
Sep 17 17:35:13.163709 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:13.163682    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv" event={"ID":"6f5a84a2-2e30-413a-823f-29cbf761cee0","Type":"ContainerStarted","Data":"58c8b8700e13fdceef37cb524fb3b1b52cedc15109ea383b94480fcec6768bab"}
Sep 17 17:35:13.165067 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:13.165042    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9tbc5" event={"ID":"57984009-dbb5-41b6-a1cb-d66608916495","Type":"ContainerStarted","Data":"dd31c9fb834ba887b508047ff3084c95b073b4a2d6100626aaad516e1d8c0356"}
Sep 17 17:35:13.166083 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:13.166062    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gtn5h"
Sep 17 17:35:13.166183 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:13.166124    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gtn5h"
Sep 17 17:35:13.183816 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:13.183767    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9tbc5" podStartSLOduration=4.714999558 podStartE2EDuration="22.183749038s" totalImagesPullingTime="17.46874948s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:13.183292925 +0000 UTC m=+24.274599700" watchObservedRunningTime="2026-09-17 17:35:13.183749038 +0000 UTC m=+24.275055802"
Sep 17 17:35:14.170582 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:14.170495    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n586" event={"ID":"c1593193-cc86-43b4-bfd5-75855a7d027d","Type":"ContainerStarted","Data":"693a020fefec134196c0bb7c41acf541acc0df42367a373f553fe430cd1c2e87"}
Sep 17 17:35:14.193624 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:14.193579    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9n586" podStartSLOduration=3.600599581 podStartE2EDuration="23.19356001s" totalImagesPullingTime="19.592960429s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:14.193349783 +0000 UTC m=+25.284656597" watchObservedRunningTime="2026-09-17 17:35:14.19356001 +0000 UTC m=+25.284866774"
Sep 17 17:35:15.034788 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:15.034754    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh98p"
Sep 17 17:35:15.034935 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:15.034793    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksb2q"
Sep 17 17:35:15.034935 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:15.034899    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh98p" podUID="ae10190f-448c-4e69-a268-85f77c85d7b6"
Sep 17 17:35:15.035049 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:15.034962    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksb2q" podUID="df33b3f5-aa93-4cb0-b1cb-3d89813c9bac"
Sep 17 17:35:15.035049 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:15.035028    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:35:15.035151 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:15.035129    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c46w4" podUID="4696b495-138f-4b5a-838b-d13ec3d8a3a6"
Sep 17 17:35:16.179162 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:16.178775    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv" event={"ID":"6f5a84a2-2e30-413a-823f-29cbf761cee0","Type":"ContainerStarted","Data":"e3e2ee3adfd545ac75e00090b0c6ac58f3ed208e5e209b65494630b0b364fac4"}
Sep 17 17:35:17.035440 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:17.035403    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh98p"
Sep 17 17:35:17.035608 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:17.035408    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksb2q"
Sep 17 17:35:17.035608 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:17.035535    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:35:17.035706 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:17.035506    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh98p" podUID="ae10190f-448c-4e69-a268-85f77c85d7b6"
Sep 17 17:35:17.035706 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:17.035611    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksb2q" podUID="df33b3f5-aa93-4cb0-b1cb-3d89813c9bac"
Sep 17 17:35:17.035706 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:17.035683    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c46w4" podUID="4696b495-138f-4b5a-838b-d13ec3d8a3a6"
Sep 17 17:35:17.182527 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:17.182497    2583 generic.go:356] "Generic (PLEG): container finished" podID="ebc5fcd3-7a3c-4922-b880-c677bdb77777" containerID="2dd944584999d5663aa98de7cf78aaefacd97763223328a68ebabad942af75b6" exitCode=0
Sep 17 17:35:17.182961 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:17.182545    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rbtw" event={"ID":"ebc5fcd3-7a3c-4922-b880-c677bdb77777","Type":"ContainerDied","Data":"2dd944584999d5663aa98de7cf78aaefacd97763223328a68ebabad942af75b6"}
Sep 17 17:35:18.189573 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:18.188943    2583 generic.go:356] "Generic (PLEG): container finished" podID="ebc5fcd3-7a3c-4922-b880-c677bdb77777" containerID="b668ca291cadc4ea6cf98b2f90844a2431f8d067332f26e5523888c16f7f8f86" exitCode=0
Sep 17 17:35:18.189573 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:18.188967    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rbtw" event={"ID":"ebc5fcd3-7a3c-4922-b880-c677bdb77777","Type":"ContainerDied","Data":"b668ca291cadc4ea6cf98b2f90844a2431f8d067332f26e5523888c16f7f8f86"}
Sep 17 17:35:19.034780 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:19.034531    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh98p"
Sep 17 17:35:19.034937 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:19.034536    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:35:19.034937 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:19.034825    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh98p" podUID="ae10190f-448c-4e69-a268-85f77c85d7b6"
Sep 17 17:35:19.034937 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:19.034537    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksb2q"
Sep 17 17:35:19.034937 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:19.034878    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c46w4" podUID="4696b495-138f-4b5a-838b-d13ec3d8a3a6"
Sep 17 17:35:19.035124 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:19.034932    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksb2q" podUID="df33b3f5-aa93-4cb0-b1cb-3d89813c9bac"
Sep 17 17:35:19.194688 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:19.194641    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv" event={"ID":"6f5a84a2-2e30-413a-823f-29cbf761cee0","Type":"ContainerStarted","Data":"b6ed88a8f29446379b82436ba450bbb11261b361dc7e830700b9082febee6257"}
Sep 17 17:35:19.195336 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:19.194990    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:35:19.197237 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:19.197207    2583 generic.go:356] "Generic (PLEG): container finished" podID="ebc5fcd3-7a3c-4922-b880-c677bdb77777" containerID="795897bf87c185fbfe148dde6f580add1076f96c60f9e543aaa5937c6bbc4e27" exitCode=0
Sep 17 17:35:19.197371 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:19.197253    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rbtw" event={"ID":"ebc5fcd3-7a3c-4922-b880-c677bdb77777","Type":"ContainerDied","Data":"795897bf87c185fbfe148dde6f580add1076f96c60f9e543aaa5937c6bbc4e27"}
Sep 17 17:35:19.209911 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:19.209891    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:35:19.225196 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:19.225157    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv" podStartSLOduration=10.167265706 podStartE2EDuration="28.225145258s" totalImagesPullingTime="18.057879552s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:19.224556994 +0000 UTC m=+30.315863786" watchObservedRunningTime="2026-09-17 17:35:19.225145258 +0000 UTC m=+30.316452023"
Sep 17 17:35:20.200716 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:20.200681    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:35:20.201124 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:20.200791    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:35:20.218031 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:20.218004    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:35:20.733762 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:20.733653    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c46w4"]
Sep 17 17:35:20.734001 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:20.733811    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:35:20.734001 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:20.733925    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c46w4" podUID="4696b495-138f-4b5a-838b-d13ec3d8a3a6"
Sep 17 17:35:20.734921 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:20.734767    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fh98p"]
Sep 17 17:35:20.734921 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:20.734908    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh98p"
Sep 17 17:35:20.735115 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:20.735060    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh98p" podUID="ae10190f-448c-4e69-a268-85f77c85d7b6"
Sep 17 17:35:20.735348 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:20.735323    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ksb2q"]
Sep 17 17:35:20.735426 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:20.735417    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksb2q"
Sep 17 17:35:20.735618 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:20.735494    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksb2q" podUID="df33b3f5-aa93-4cb0-b1cb-3d89813c9bac"
Sep 17 17:35:22.034863 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:22.034832    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh98p"
Sep 17 17:35:22.035308 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:22.034942    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh98p" podUID="ae10190f-448c-4e69-a268-85f77c85d7b6"
Sep 17 17:35:23.034960 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:23.034926    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksb2q"
Sep 17 17:35:23.035524 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:23.034926    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:35:23.035524 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:23.035057    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksb2q" podUID="df33b3f5-aa93-4cb0-b1cb-3d89813c9bac"
Sep 17 17:35:23.035524 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:23.035147    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c46w4" podUID="4696b495-138f-4b5a-838b-d13ec3d8a3a6"
Sep 17 17:35:24.035102 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:24.034936    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh98p"
Sep 17 17:35:24.035453 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:24.035176    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fh98p" podUID="ae10190f-448c-4e69-a268-85f77c85d7b6"
Sep 17 17:35:24.211444 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:24.211414    2583 generic.go:356] "Generic (PLEG): container finished" podID="ebc5fcd3-7a3c-4922-b880-c677bdb77777" containerID="7e1d930edef57af0a0c67cae1270552427c0e3a104cb2dd7844995def1ee8979" exitCode=0
Sep 17 17:35:24.211602 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:24.211456    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rbtw" event={"ID":"ebc5fcd3-7a3c-4922-b880-c677bdb77777","Type":"ContainerDied","Data":"7e1d930edef57af0a0c67cae1270552427c0e3a104cb2dd7844995def1ee8979"}
Sep 17 17:35:24.643133 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:24.643106    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4696b495-138f-4b5a-838b-d13ec3d8a3a6-original-pull-secret\") pod \"global-pull-secret-syncer-c46w4\" (UID: \"4696b495-138f-4b5a-838b-d13ec3d8a3a6\") " pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:35:24.643258 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:24.643163    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae10190f-448c-4e69-a268-85f77c85d7b6-metrics-certs\") pod \"network-metrics-daemon-fh98p\" (UID: \"ae10190f-448c-4e69-a268-85f77c85d7b6\") " pod="openshift-multus/network-metrics-daemon-fh98p"
Sep 17 17:35:24.643300 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:24.643254    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:24.643300 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:24.643257    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:24.643361 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:24.643318    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae10190f-448c-4e69-a268-85f77c85d7b6-metrics-certs podName:ae10190f-448c-4e69-a268-85f77c85d7b6 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:56.643300302 +0000 UTC m=+67.734607045 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ae10190f-448c-4e69-a268-85f77c85d7b6-metrics-certs") pod "network-metrics-daemon-fh98p" (UID: "ae10190f-448c-4e69-a268-85f77c85d7b6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:24.643361 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:24.643331    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4696b495-138f-4b5a-838b-d13ec3d8a3a6-original-pull-secret podName:4696b495-138f-4b5a-838b-d13ec3d8a3a6 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:56.643325204 +0000 UTC m=+67.734631946 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4696b495-138f-4b5a-838b-d13ec3d8a3a6-original-pull-secret") pod "global-pull-secret-syncer-c46w4" (UID: "4696b495-138f-4b5a-838b-d13ec3d8a3a6") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:24.844653 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:24.844607    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87775\" (UniqueName: \"kubernetes.io/projected/df33b3f5-aa93-4cb0-b1cb-3d89813c9bac-kube-api-access-87775\") pod \"network-check-target-ksb2q\" (UID: \"df33b3f5-aa93-4cb0-b1cb-3d89813c9bac\") " pod="openshift-network-diagnostics/network-check-target-ksb2q"
Sep 17 17:35:24.844873 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:24.844785    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:35:24.844873 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:24.844806    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:35:24.844873 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:24.844818    2583 projected.go:196] Error preparing data for projected volume kube-api-access-87775 for pod openshift-network-diagnostics/network-check-target-ksb2q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:24.845026 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:24.844880    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df33b3f5-aa93-4cb0-b1cb-3d89813c9bac-kube-api-access-87775 podName:df33b3f5-aa93-4cb0-b1cb-3d89813c9bac nodeName:}" failed. No retries permitted until 2026-09-17 17:35:56.844862799 +0000 UTC m=+67.936169547 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-87775" (UniqueName: "kubernetes.io/projected/df33b3f5-aa93-4cb0-b1cb-3d89813c9bac-kube-api-access-87775") pod "network-check-target-ksb2q" (UID: "df33b3f5-aa93-4cb0-b1cb-3d89813c9bac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:25.034731 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.034710    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:35:25.034839 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:25.034804    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c46w4" podUID="4696b495-138f-4b5a-838b-d13ec3d8a3a6"
Sep 17 17:35:25.034879 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.034864    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksb2q"
Sep 17 17:35:25.034966 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:25.034949    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ksb2q" podUID="df33b3f5-aa93-4cb0-b1cb-3d89813c9bac"
Sep 17 17:35:25.216880 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.216849    2583 generic.go:356] "Generic (PLEG): container finished" podID="ebc5fcd3-7a3c-4922-b880-c677bdb77777" containerID="e175f32f69e17a25a969eadc2fb9637cdfecce52a7b5c603b9bb897fe8cc4cb0" exitCode=0
Sep 17 17:35:25.217306 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.216895    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rbtw" event={"ID":"ebc5fcd3-7a3c-4922-b880-c677bdb77777","Type":"ContainerDied","Data":"e175f32f69e17a25a969eadc2fb9637cdfecce52a7b5c603b9bb897fe8cc4cb0"}
Sep 17 17:35:25.301436 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.301413    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-30.ec2.internal" event="NodeReady"
Sep 17 17:35:25.301563 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.301551    2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 17:35:25.346702 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.346678    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-d6z4r"]
Sep 17 17:35:25.349562 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.349530    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d6z4r"
Sep 17 17:35:25.349684 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.349533    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-q5nq9"]
Sep 17 17:35:25.351513 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.351497    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 17:35:25.351786 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.351759    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 17:35:25.351966 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.351951    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dng7x\""
Sep 17 17:35:25.352404 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.352388    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-q5nq9"
Sep 17 17:35:25.354622 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.354608    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 17:35:25.354761 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.354680    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 17:35:25.354927 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.354897    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wr4qk\""
Sep 17 17:35:25.354927 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.354918    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 17:35:25.358651 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.358633    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d6z4r"]
Sep 17 17:35:25.361206 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.361188    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-q5nq9"]
Sep 17 17:35:25.449047 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.449022    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j22mb\" (UniqueName: \"kubernetes.io/projected/dad536d7-d27e-4d63-96e9-a6654126532d-kube-api-access-j22mb\") pod \"dns-default-d6z4r\" (UID: \"dad536d7-d27e-4d63-96e9-a6654126532d\") " pod="openshift-dns/dns-default-d6z4r"
Sep 17 17:35:25.449176 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.449058    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqssq\" (UniqueName: \"kubernetes.io/projected/be8c6ba8-3ad9-4988-b871-01c0d8807b10-kube-api-access-jqssq\") pod \"ingress-canary-q5nq9\" (UID: \"be8c6ba8-3ad9-4988-b871-01c0d8807b10\") " pod="openshift-ingress-canary/ingress-canary-q5nq9"
Sep 17 17:35:25.449176 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.449122    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad536d7-d27e-4d63-96e9-a6654126532d-tmp-dir\") pod \"dns-default-d6z4r\" (UID: \"dad536d7-d27e-4d63-96e9-a6654126532d\") " pod="openshift-dns/dns-default-d6z4r"
Sep 17 17:35:25.449176 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.449163    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dad536d7-d27e-4d63-96e9-a6654126532d-metrics-tls\") pod \"dns-default-d6z4r\" (UID: \"dad536d7-d27e-4d63-96e9-a6654126532d\") " pod="openshift-dns/dns-default-d6z4r"
Sep 17 17:35:25.449313 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.449184    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be8c6ba8-3ad9-4988-b871-01c0d8807b10-cert\") pod \"ingress-canary-q5nq9\" (UID: \"be8c6ba8-3ad9-4988-b871-01c0d8807b10\") " pod="openshift-ingress-canary/ingress-canary-q5nq9"
Sep 17 17:35:25.449313 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.449231    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dad536d7-d27e-4d63-96e9-a6654126532d-config-volume\") pod \"dns-default-d6z4r\" (UID: \"dad536d7-d27e-4d63-96e9-a6654126532d\") " pod="openshift-dns/dns-default-d6z4r"
Sep 17 17:35:25.550403 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.550330    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad536d7-d27e-4d63-96e9-a6654126532d-tmp-dir\") pod \"dns-default-d6z4r\" (UID: \"dad536d7-d27e-4d63-96e9-a6654126532d\") " pod="openshift-dns/dns-default-d6z4r"
Sep 17 17:35:25.550403 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.550374    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dad536d7-d27e-4d63-96e9-a6654126532d-metrics-tls\") pod \"dns-default-d6z4r\" (UID: \"dad536d7-d27e-4d63-96e9-a6654126532d\") " pod="openshift-dns/dns-default-d6z4r"
Sep 17 17:35:25.550403 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.550395    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be8c6ba8-3ad9-4988-b871-01c0d8807b10-cert\") pod \"ingress-canary-q5nq9\" (UID: \"be8c6ba8-3ad9-4988-b871-01c0d8807b10\") " pod="openshift-ingress-canary/ingress-canary-q5nq9"
Sep 17 17:35:25.550638 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.550421    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dad536d7-d27e-4d63-96e9-a6654126532d-config-volume\") pod \"dns-default-d6z4r\" (UID: \"dad536d7-d27e-4d63-96e9-a6654126532d\") " pod="openshift-dns/dns-default-d6z4r"
Sep 17 17:35:25.550638 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.550443    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j22mb\" (UniqueName: \"kubernetes.io/projected/dad536d7-d27e-4d63-96e9-a6654126532d-kube-api-access-j22mb\") pod \"dns-default-d6z4r\" (UID: \"dad536d7-d27e-4d63-96e9-a6654126532d\") " pod="openshift-dns/dns-default-d6z4r"
Sep 17 17:35:25.550638 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.550463    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqssq\" (UniqueName: \"kubernetes.io/projected/be8c6ba8-3ad9-4988-b871-01c0d8807b10-kube-api-access-jqssq\") pod \"ingress-canary-q5nq9\" (UID: \"be8c6ba8-3ad9-4988-b871-01c0d8807b10\") " pod="openshift-ingress-canary/ingress-canary-q5nq9"
Sep 17 17:35:25.550638 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:25.550517    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:25.550638 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:25.550582    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dad536d7-d27e-4d63-96e9-a6654126532d-metrics-tls podName:dad536d7-d27e-4d63-96e9-a6654126532d nodeName:}" failed. No retries permitted until 2026-09-17 17:35:26.050562496 +0000 UTC m=+37.141869255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dad536d7-d27e-4d63-96e9-a6654126532d-metrics-tls") pod "dns-default-d6z4r" (UID: "dad536d7-d27e-4d63-96e9-a6654126532d") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:25.550886 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.550738    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dad536d7-d27e-4d63-96e9-a6654126532d-tmp-dir\") pod \"dns-default-d6z4r\" (UID: \"dad536d7-d27e-4d63-96e9-a6654126532d\") " pod="openshift-dns/dns-default-d6z4r"
Sep 17 17:35:25.550992 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.550975    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dad536d7-d27e-4d63-96e9-a6654126532d-config-volume\") pod \"dns-default-d6z4r\" (UID: \"dad536d7-d27e-4d63-96e9-a6654126532d\") " pod="openshift-dns/dns-default-d6z4r"
Sep 17 17:35:25.554444 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.554418    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be8c6ba8-3ad9-4988-b871-01c0d8807b10-cert\") pod \"ingress-canary-q5nq9\" (UID: \"be8c6ba8-3ad9-4988-b871-01c0d8807b10\") " pod="openshift-ingress-canary/ingress-canary-q5nq9"
Sep 17 17:35:25.559301 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.559275    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j22mb\" (UniqueName: \"kubernetes.io/projected/dad536d7-d27e-4d63-96e9-a6654126532d-kube-api-access-j22mb\") pod \"dns-default-d6z4r\" (UID: \"dad536d7-d27e-4d63-96e9-a6654126532d\") " pod="openshift-dns/dns-default-d6z4r"
Sep 17 17:35:25.559404 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.559287    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqssq\" (UniqueName: \"kubernetes.io/projected/be8c6ba8-3ad9-4988-b871-01c0d8807b10-kube-api-access-jqssq\") pod \"ingress-canary-q5nq9\" (UID: \"be8c6ba8-3ad9-4988-b871-01c0d8807b10\") " pod="openshift-ingress-canary/ingress-canary-q5nq9"
Sep 17 17:35:25.665491 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.665442    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-q5nq9"
Sep 17 17:35:25.944531 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:25.944495    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-q5nq9"]
Sep 17 17:35:25.949142 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:35:25.949027    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe8c6ba8_3ad9_4988_b871_01c0d8807b10.slice/crio-366dc94c34654a2c42f57bffd9a057fae568174c365ba7b3a88a4d3103c43dea WatchSource:0}: Error finding container 366dc94c34654a2c42f57bffd9a057fae568174c365ba7b3a88a4d3103c43dea: Status 404 returned error can't find the container with id 366dc94c34654a2c42f57bffd9a057fae568174c365ba7b3a88a4d3103c43dea
Sep 17 17:35:26.034737 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:26.034714    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh98p"
Sep 17 17:35:26.037487 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:26.037463    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:35:26.037621 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:26.037506    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j84n5\""
Sep 17 17:35:26.054475 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:26.054451    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dad536d7-d27e-4d63-96e9-a6654126532d-metrics-tls\") pod \"dns-default-d6z4r\" (UID: \"dad536d7-d27e-4d63-96e9-a6654126532d\") " pod="openshift-dns/dns-default-d6z4r"
Sep 17 17:35:26.054574 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:26.054562    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:26.054627 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:26.054618    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dad536d7-d27e-4d63-96e9-a6654126532d-metrics-tls podName:dad536d7-d27e-4d63-96e9-a6654126532d nodeName:}" failed. No retries permitted until 2026-09-17 17:35:27.054605291 +0000 UTC m=+38.145912034 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dad536d7-d27e-4d63-96e9-a6654126532d-metrics-tls") pod "dns-default-d6z4r" (UID: "dad536d7-d27e-4d63-96e9-a6654126532d") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:26.220582 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:26.220474    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-q5nq9" event={"ID":"be8c6ba8-3ad9-4988-b871-01c0d8807b10","Type":"ContainerStarted","Data":"366dc94c34654a2c42f57bffd9a057fae568174c365ba7b3a88a4d3103c43dea"}
Sep 17 17:35:26.223645 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:26.223617    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7rbtw" event={"ID":"ebc5fcd3-7a3c-4922-b880-c677bdb77777","Type":"ContainerStarted","Data":"2935723972c0e7de2a9846421edaed84a723e917a3468012296e85940c22b501"}
Sep 17 17:35:26.247616 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:26.247577    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7rbtw" podStartSLOduration=8.53108849 podStartE2EDuration="35.247562604s" totalImagesPullingTime="26.716474114s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:26.247505933 +0000 UTC m=+37.338812712" watchObservedRunningTime="2026-09-17 17:35:26.247562604 +0000 UTC m=+37.338869368"
Sep 17 17:35:27.034465 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:27.034431    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksb2q"
Sep 17 17:35:27.034808 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:27.034783    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:35:27.037212 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:27.037186    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:35:27.037945 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:27.037924    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:35:27.038337 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:27.038073    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:35:27.038337 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:27.038080    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vwwqh\""
Sep 17 17:35:27.061831 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:27.061808    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dad536d7-d27e-4d63-96e9-a6654126532d-metrics-tls\") pod \"dns-default-d6z4r\" (UID: \"dad536d7-d27e-4d63-96e9-a6654126532d\") " pod="openshift-dns/dns-default-d6z4r"
Sep 17 17:35:27.061983 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:27.061966    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:27.062091 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:27.062032    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dad536d7-d27e-4d63-96e9-a6654126532d-metrics-tls podName:dad536d7-d27e-4d63-96e9-a6654126532d nodeName:}" failed. No retries permitted until 2026-09-17 17:35:29.062015572 +0000 UTC m=+40.153322316 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dad536d7-d27e-4d63-96e9-a6654126532d-metrics-tls") pod "dns-default-d6z4r" (UID: "dad536d7-d27e-4d63-96e9-a6654126532d") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:29.075372 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:29.075342    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dad536d7-d27e-4d63-96e9-a6654126532d-metrics-tls\") pod \"dns-default-d6z4r\" (UID: \"dad536d7-d27e-4d63-96e9-a6654126532d\") " pod="openshift-dns/dns-default-d6z4r"
Sep 17 17:35:29.075938 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:29.075463    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:29.075938 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:29.075535    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dad536d7-d27e-4d63-96e9-a6654126532d-metrics-tls podName:dad536d7-d27e-4d63-96e9-a6654126532d nodeName:}" failed. No retries permitted until 2026-09-17 17:35:33.075514475 +0000 UTC m=+44.166821223 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dad536d7-d27e-4d63-96e9-a6654126532d-metrics-tls") pod "dns-default-d6z4r" (UID: "dad536d7-d27e-4d63-96e9-a6654126532d") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:29.234392 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:29.234359    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-q5nq9" event={"ID":"be8c6ba8-3ad9-4988-b871-01c0d8807b10","Type":"ContainerStarted","Data":"62e2eb0cf969a193f96f70e92f0f432992f1911350e43ebeb12619882fb6c472"}
Sep 17 17:35:29.255730 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:29.255684    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-q5nq9" podStartSLOduration=1.688746944 podStartE2EDuration="4.255654717s" totalImagesPullingTime="2.566907773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:29.255159884 +0000 UTC m=+40.346466663" watchObservedRunningTime="2026-09-17 17:35:29.255654717 +0000 UTC m=+40.346961481"
Sep 17 17:35:33.102574 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:33.102529    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dad536d7-d27e-4d63-96e9-a6654126532d-metrics-tls\") pod \"dns-default-d6z4r\" (UID: \"dad536d7-d27e-4d63-96e9-a6654126532d\") " pod="openshift-dns/dns-default-d6z4r"
Sep 17 17:35:33.102986 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:33.102698    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:33.102986 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:33.102764    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dad536d7-d27e-4d63-96e9-a6654126532d-metrics-tls podName:dad536d7-d27e-4d63-96e9-a6654126532d nodeName:}" failed. No retries permitted until 2026-09-17 17:35:41.102747379 +0000 UTC m=+52.194054121 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dad536d7-d27e-4d63-96e9-a6654126532d-metrics-tls") pod "dns-default-d6z4r" (UID: "dad536d7-d27e-4d63-96e9-a6654126532d") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:41.152228 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:41.152189    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dad536d7-d27e-4d63-96e9-a6654126532d-metrics-tls\") pod \"dns-default-d6z4r\" (UID: \"dad536d7-d27e-4d63-96e9-a6654126532d\") " pod="openshift-dns/dns-default-d6z4r"
Sep 17 17:35:41.152827 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:41.152389    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:41.152827 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:41.152484    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dad536d7-d27e-4d63-96e9-a6654126532d-metrics-tls podName:dad536d7-d27e-4d63-96e9-a6654126532d nodeName:}" failed. No retries permitted until 2026-09-17 17:35:57.152461735 +0000 UTC m=+68.243768483 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dad536d7-d27e-4d63-96e9-a6654126532d-metrics-tls") pod "dns-default-d6z4r" (UID: "dad536d7-d27e-4d63-96e9-a6654126532d") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:52.215372 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:52.215344    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9fhwv"
Sep 17 17:35:56.666143 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:56.666094    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae10190f-448c-4e69-a268-85f77c85d7b6-metrics-certs\") pod \"network-metrics-daemon-fh98p\" (UID: \"ae10190f-448c-4e69-a268-85f77c85d7b6\") " pod="openshift-multus/network-metrics-daemon-fh98p"
Sep 17 17:35:56.666583 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:56.666171    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4696b495-138f-4b5a-838b-d13ec3d8a3a6-original-pull-secret\") pod \"global-pull-secret-syncer-c46w4\" (UID: \"4696b495-138f-4b5a-838b-d13ec3d8a3a6\") " pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:35:56.675696 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:56.675677    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:35:56.675752 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:56.675704    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:35:56.676722 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:56.676703    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 17:35:56.676791 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:56.676776    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae10190f-448c-4e69-a268-85f77c85d7b6-metrics-certs podName:ae10190f-448c-4e69-a268-85f77c85d7b6 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:00.676753155 +0000 UTC m=+131.768059904 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ae10190f-448c-4e69-a268-85f77c85d7b6-metrics-certs") pod "network-metrics-daemon-fh98p" (UID: "ae10190f-448c-4e69-a268-85f77c85d7b6") : secret "metrics-daemon-secret" not found
Sep 17 17:35:56.679767 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:56.679744    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4696b495-138f-4b5a-838b-d13ec3d8a3a6-original-pull-secret\") pod \"global-pull-secret-syncer-c46w4\" (UID: \"4696b495-138f-4b5a-838b-d13ec3d8a3a6\") " pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:35:56.753331 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:56.753301    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c46w4"
Sep 17 17:35:56.867481 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:56.867438    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87775\" (UniqueName: \"kubernetes.io/projected/df33b3f5-aa93-4cb0-b1cb-3d89813c9bac-kube-api-access-87775\") pod \"network-check-target-ksb2q\" (UID: \"df33b3f5-aa93-4cb0-b1cb-3d89813c9bac\") " pod="openshift-network-diagnostics/network-check-target-ksb2q"
Sep 17 17:35:56.870105 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:56.870088    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:35:56.880507 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:56.880488    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:35:56.890394 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:56.890374    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87775\" (UniqueName: \"kubernetes.io/projected/df33b3f5-aa93-4cb0-b1cb-3d89813c9bac-kube-api-access-87775\") pod \"network-check-target-ksb2q\" (UID: \"df33b3f5-aa93-4cb0-b1cb-3d89813c9bac\") " pod="openshift-network-diagnostics/network-check-target-ksb2q"
Sep 17 17:35:57.022697 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:57.022615    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c46w4"]
Sep 17 17:35:57.026367 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:35:57.026335    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4696b495_138f_4b5a_838b_d13ec3d8a3a6.slice/crio-7f454e9636cc18ed31f07e24a66a9cac8d111311f8a8ce7145c06dd77477dc91 WatchSource:0}: Error finding container 7f454e9636cc18ed31f07e24a66a9cac8d111311f8a8ce7145c06dd77477dc91: Status 404 returned error can't find the container with id 7f454e9636cc18ed31f07e24a66a9cac8d111311f8a8ce7145c06dd77477dc91
Sep 17 17:35:57.050451 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:57.050433    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vwwqh\""
Sep 17 17:35:57.057722 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:57.057697    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ksb2q"
Sep 17 17:35:57.170378 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:57.170348    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dad536d7-d27e-4d63-96e9-a6654126532d-metrics-tls\") pod \"dns-default-d6z4r\" (UID: \"dad536d7-d27e-4d63-96e9-a6654126532d\") " pod="openshift-dns/dns-default-d6z4r"
Sep 17 17:35:57.170528 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:57.170463    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:57.170528 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:35:57.170524    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dad536d7-d27e-4d63-96e9-a6654126532d-metrics-tls podName:dad536d7-d27e-4d63-96e9-a6654126532d nodeName:}" failed. No retries permitted until 2026-09-17 17:36:29.170509957 +0000 UTC m=+100.261816700 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dad536d7-d27e-4d63-96e9-a6654126532d-metrics-tls") pod "dns-default-d6z4r" (UID: "dad536d7-d27e-4d63-96e9-a6654126532d") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:57.299481 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:57.299404    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c46w4" event={"ID":"4696b495-138f-4b5a-838b-d13ec3d8a3a6","Type":"ContainerStarted","Data":"7f454e9636cc18ed31f07e24a66a9cac8d111311f8a8ce7145c06dd77477dc91"}
Sep 17 17:35:57.327579 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:57.327549    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ksb2q"]
Sep 17 17:35:57.330667 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:35:57.330632    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf33b3f5_aa93_4cb0_b1cb_3d89813c9bac.slice/crio-997f51e0d16af9e6318b1cc33de643a1304d43e27c8bdeb4ce1bdac6aa06fe14 WatchSource:0}: Error finding container 997f51e0d16af9e6318b1cc33de643a1304d43e27c8bdeb4ce1bdac6aa06fe14: Status 404 returned error can't find the container with id 997f51e0d16af9e6318b1cc33de643a1304d43e27c8bdeb4ce1bdac6aa06fe14
Sep 17 17:35:58.304177 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:35:58.304138    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ksb2q" event={"ID":"df33b3f5-aa93-4cb0-b1cb-3d89813c9bac","Type":"ContainerStarted","Data":"997f51e0d16af9e6318b1cc33de643a1304d43e27c8bdeb4ce1bdac6aa06fe14"}
Sep 17 17:36:02.315575 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:02.315448    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ksb2q" event={"ID":"df33b3f5-aa93-4cb0-b1cb-3d89813c9bac","Type":"ContainerStarted","Data":"e9481742e10db462095e8f31ca4f7046ac4e331bbaa3a23bc8a97c06aea9fb89"}
Sep 17 17:36:02.315575 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:02.315516    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ksb2q"
Sep 17 17:36:02.316917 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:02.316894    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c46w4" event={"ID":"4696b495-138f-4b5a-838b-d13ec3d8a3a6","Type":"ContainerStarted","Data":"e2310ac8955c6bf733302172db39b0893ef2bd074e39fee9e4d818d5f82209bf"}
Sep 17 17:36:02.337352 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:02.337287    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ksb2q" podStartSLOduration=67.132797994 podStartE2EDuration="1m11.337268742s" totalImagesPullingTime="4.204470748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:02.336868188 +0000 UTC m=+73.428174972" watchObservedRunningTime="2026-09-17 17:36:02.337268742 +0000 UTC m=+73.428575564"
Sep 17 17:36:23.949792 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:23.949741    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-c46w4" podStartSLOduration=88.437468843 podStartE2EDuration="1m32.949706463s" totalImagesPullingTime="4.51223762s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:02.359383139 +0000 UTC m=+73.450689903" watchObservedRunningTime="2026-09-17 17:36:23.949706463 +0000 UTC m=+95.041013210"
Sep 17 17:36:23.950591 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:23.950571    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ftqq"]
Sep 17 17:36:23.952832 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:23.952817    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ftqq"
Sep 17 17:36:23.954740 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:23.954717    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-hx6xd"]
Sep 17 17:36:23.955416 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:23.955399    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 17 17:36:23.956405 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:23.956353    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-hx6xd"
Sep 17 17:36:23.956603 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:23.956578    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 17:36:23.956728 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:23.956711    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-268t5\""
Sep 17 17:36:23.957521 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:23.957506    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 17:36:23.957572 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:23.957539    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 17 17:36:23.958638 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:23.958628    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 17 17:36:23.958722 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:23.958711    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-9vwcq\""
Sep 17 17:36:23.959232 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:23.959216    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:36:23.968389 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:23.968356    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ftqq"]
Sep 17 17:36:23.969161 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:23.969142    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-hx6xd"]
Sep 17 17:36:24.048353 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:24.048329    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f11e6c11-e031-485d-8ffa-f71b00f75347-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-5ftqq\" (UID: \"f11e6c11-e031-485d-8ffa-f71b00f75347\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ftqq"
Sep 17 17:36:24.048478 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:24.048377    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7s7b\" (UniqueName: \"kubernetes.io/projected/f11e6c11-e031-485d-8ffa-f71b00f75347-kube-api-access-x7s7b\") pod \"cluster-monitoring-operator-847b8bd8d7-5ftqq\" (UID: \"f11e6c11-e031-485d-8ffa-f71b00f75347\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ftqq"
Sep 17 17:36:24.048478 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:24.048430    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f11e6c11-e031-485d-8ffa-f71b00f75347-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-5ftqq\" (UID: \"f11e6c11-e031-485d-8ffa-f71b00f75347\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ftqq"
Sep 17 17:36:24.048478 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:24.048463    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnf9v\" (UniqueName: \"kubernetes.io/projected/4893c222-3bba-4071-ae44-3f04bd4c523c-kube-api-access-cnf9v\") pod \"volume-data-source-validator-5c4db8bdf9-hx6xd\" (UID: \"4893c222-3bba-4071-ae44-3f04bd4c523c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-hx6xd"
Sep 17 17:36:24.149737 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:24.149702    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f11e6c11-e031-485d-8ffa-f71b00f75347-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-5ftqq\" (UID: \"f11e6c11-e031-485d-8ffa-f71b00f75347\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ftqq"
Sep 17 17:36:24.149737 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:24.149738    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnf9v\" (UniqueName: \"kubernetes.io/projected/4893c222-3bba-4071-ae44-3f04bd4c523c-kube-api-access-cnf9v\") pod \"volume-data-source-validator-5c4db8bdf9-hx6xd\" (UID: \"4893c222-3bba-4071-ae44-3f04bd4c523c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-hx6xd"
Sep 17 17:36:24.150043 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:24.149768    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f11e6c11-e031-485d-8ffa-f71b00f75347-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-5ftqq\" (UID: \"f11e6c11-e031-485d-8ffa-f71b00f75347\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ftqq"
Sep 17 17:36:24.150043 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:24.149853    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:24.150043 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:24.149904    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7s7b\" (UniqueName: \"kubernetes.io/projected/f11e6c11-e031-485d-8ffa-f71b00f75347-kube-api-access-x7s7b\") pod \"cluster-monitoring-operator-847b8bd8d7-5ftqq\" (UID: \"f11e6c11-e031-485d-8ffa-f71b00f75347\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ftqq"
Sep 17 17:36:24.150043 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:24.149972    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f11e6c11-e031-485d-8ffa-f71b00f75347-cluster-monitoring-operator-tls podName:f11e6c11-e031-485d-8ffa-f71b00f75347 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:24.649950457 +0000 UTC m=+95.741257201 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f11e6c11-e031-485d-8ffa-f71b00f75347-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-5ftqq" (UID: "f11e6c11-e031-485d-8ffa-f71b00f75347") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:24.150384 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:24.150367    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f11e6c11-e031-485d-8ffa-f71b00f75347-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-5ftqq\" (UID: \"f11e6c11-e031-485d-8ffa-f71b00f75347\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ftqq"
Sep 17 17:36:24.171258 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:24.171235    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7s7b\" (UniqueName: \"kubernetes.io/projected/f11e6c11-e031-485d-8ffa-f71b00f75347-kube-api-access-x7s7b\") pod \"cluster-monitoring-operator-847b8bd8d7-5ftqq\" (UID: \"f11e6c11-e031-485d-8ffa-f71b00f75347\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ftqq"
Sep 17 17:36:24.175265 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:24.175247    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnf9v\" (UniqueName: \"kubernetes.io/projected/4893c222-3bba-4071-ae44-3f04bd4c523c-kube-api-access-cnf9v\") pod \"volume-data-source-validator-5c4db8bdf9-hx6xd\" (UID: \"4893c222-3bba-4071-ae44-3f04bd4c523c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-hx6xd"
Sep 17 17:36:24.267551 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:24.267470    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-hx6xd"
Sep 17 17:36:24.535018 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:24.534947    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-hx6xd"]
Sep 17 17:36:24.538060 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:36:24.538022    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4893c222_3bba_4071_ae44_3f04bd4c523c.slice/crio-347e3b7ee7981576f154f4a3105e95969654ead282c66394cf942a57090e364c WatchSource:0}: Error finding container 347e3b7ee7981576f154f4a3105e95969654ead282c66394cf942a57090e364c: Status 404 returned error can't find the container with id 347e3b7ee7981576f154f4a3105e95969654ead282c66394cf942a57090e364c
Sep 17 17:36:24.653997 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:24.653963    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f11e6c11-e031-485d-8ffa-f71b00f75347-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-5ftqq\" (UID: \"f11e6c11-e031-485d-8ffa-f71b00f75347\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ftqq"
Sep 17 17:36:24.654161 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:24.654083    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:24.654161 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:24.654153    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f11e6c11-e031-485d-8ffa-f71b00f75347-cluster-monitoring-operator-tls podName:f11e6c11-e031-485d-8ffa-f71b00f75347 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:25.65413787 +0000 UTC m=+96.745444613 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f11e6c11-e031-485d-8ffa-f71b00f75347-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-5ftqq" (UID: "f11e6c11-e031-485d-8ffa-f71b00f75347") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:25.226434 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.226399    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7bd7db6998-stqrm"]
Sep 17 17:36:25.227995 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.227978    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:36:25.231996 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.231974    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 17:36:25.232133 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.232016    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 17:36:25.232133 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.232083    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-55z8g\""
Sep 17 17:36:25.232307 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.232293    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 17:36:25.240428 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.240404    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 17:36:25.245080 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.245046    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bd7db6998-stqrm"]
Sep 17 17:36:25.258228 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.258203    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff45a922-492d-4d9b-9039-c3a7759481a9-ca-trust-extracted\") pod \"image-registry-7bd7db6998-stqrm\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") " pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:36:25.258345 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.258240    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk97h\" (UniqueName: \"kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-kube-api-access-tk97h\") pod \"image-registry-7bd7db6998-stqrm\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") " pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:36:25.258393 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.258344    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff45a922-492d-4d9b-9039-c3a7759481a9-image-registry-private-configuration\") pod \"image-registry-7bd7db6998-stqrm\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") " pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:36:25.258393 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.258376    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff45a922-492d-4d9b-9039-c3a7759481a9-trusted-ca\") pod \"image-registry-7bd7db6998-stqrm\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") " pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:36:25.258461 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.258398    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff45a922-492d-4d9b-9039-c3a7759481a9-installation-pull-secrets\") pod \"image-registry-7bd7db6998-stqrm\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") " pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:36:25.258497 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.258474    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-registry-tls\") pod \"image-registry-7bd7db6998-stqrm\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") " pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:36:25.258541 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.258517    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff45a922-492d-4d9b-9039-c3a7759481a9-registry-certificates\") pod \"image-registry-7bd7db6998-stqrm\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") " pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:36:25.258588 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.258540    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-bound-sa-token\") pod \"image-registry-7bd7db6998-stqrm\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") " pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:36:25.359197 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.359158    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-registry-tls\") pod \"image-registry-7bd7db6998-stqrm\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") " pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:36:25.359390 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.359218    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff45a922-492d-4d9b-9039-c3a7759481a9-registry-certificates\") pod \"image-registry-7bd7db6998-stqrm\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") " pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:36:25.359390 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:25.359338    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:36:25.359390 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:25.359358    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bd7db6998-stqrm: secret "image-registry-tls" not found
Sep 17 17:36:25.359545 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.359349    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-bound-sa-token\") pod \"image-registry-7bd7db6998-stqrm\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") " pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:36:25.359545 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:25.359413    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-registry-tls podName:ff45a922-492d-4d9b-9039-c3a7759481a9 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:25.859396467 +0000 UTC m=+96.950703216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-registry-tls") pod "image-registry-7bd7db6998-stqrm" (UID: "ff45a922-492d-4d9b-9039-c3a7759481a9") : secret "image-registry-tls" not found
Sep 17 17:36:25.359545 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.359497    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff45a922-492d-4d9b-9039-c3a7759481a9-ca-trust-extracted\") pod \"image-registry-7bd7db6998-stqrm\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") " pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:36:25.359545 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.359524    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk97h\" (UniqueName: \"kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-kube-api-access-tk97h\") pod \"image-registry-7bd7db6998-stqrm\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") " pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:36:25.359777 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.359562    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff45a922-492d-4d9b-9039-c3a7759481a9-image-registry-private-configuration\") pod \"image-registry-7bd7db6998-stqrm\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") " pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:36:25.359777 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.359624    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff45a922-492d-4d9b-9039-c3a7759481a9-trusted-ca\") pod \"image-registry-7bd7db6998-stqrm\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") " pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:36:25.359777 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.359647    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff45a922-492d-4d9b-9039-c3a7759481a9-installation-pull-secrets\") pod \"image-registry-7bd7db6998-stqrm\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") " pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:36:25.359904 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.359826    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff45a922-492d-4d9b-9039-c3a7759481a9-registry-certificates\") pod \"image-registry-7bd7db6998-stqrm\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") " pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:36:25.360358 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.360326    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff45a922-492d-4d9b-9039-c3a7759481a9-ca-trust-extracted\") pod \"image-registry-7bd7db6998-stqrm\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") " pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:36:25.361052 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.361032    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff45a922-492d-4d9b-9039-c3a7759481a9-trusted-ca\") pod \"image-registry-7bd7db6998-stqrm\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") " pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:36:25.362276 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.362255    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff45a922-492d-4d9b-9039-c3a7759481a9-installation-pull-secrets\") pod \"image-registry-7bd7db6998-stqrm\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") " pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:36:25.362372 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.362294    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff45a922-492d-4d9b-9039-c3a7759481a9-image-registry-private-configuration\") pod \"image-registry-7bd7db6998-stqrm\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") " pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:36:25.368675 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.368637    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk97h\" (UniqueName: \"kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-kube-api-access-tk97h\") pod \"image-registry-7bd7db6998-stqrm\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") " pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:36:25.369037 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.369019    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-bound-sa-token\") pod \"image-registry-7bd7db6998-stqrm\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") " pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:36:25.371168 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.371134    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-hx6xd" event={"ID":"4893c222-3bba-4071-ae44-3f04bd4c523c","Type":"ContainerStarted","Data":"347e3b7ee7981576f154f4a3105e95969654ead282c66394cf942a57090e364c"}
Sep 17 17:36:25.661969 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.661923    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f11e6c11-e031-485d-8ffa-f71b00f75347-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-5ftqq\" (UID: \"f11e6c11-e031-485d-8ffa-f71b00f75347\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ftqq"
Sep 17 17:36:25.662169 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:25.662085    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:25.662169 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:25.662163    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f11e6c11-e031-485d-8ffa-f71b00f75347-cluster-monitoring-operator-tls podName:f11e6c11-e031-485d-8ffa-f71b00f75347 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:27.662144005 +0000 UTC m=+98.753450755 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f11e6c11-e031-485d-8ffa-f71b00f75347-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-5ftqq" (UID: "f11e6c11-e031-485d-8ffa-f71b00f75347") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:25.863573 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:25.863549    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-registry-tls\") pod \"image-registry-7bd7db6998-stqrm\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") " pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:36:25.863723 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:25.863678    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:36:25.863723 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:25.863690    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bd7db6998-stqrm: secret "image-registry-tls" not found
Sep 17 17:36:25.863796 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:25.863736    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-registry-tls podName:ff45a922-492d-4d9b-9039-c3a7759481a9 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:26.86372405 +0000 UTC m=+97.955030793 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-registry-tls") pod "image-registry-7bd7db6998-stqrm" (UID: "ff45a922-492d-4d9b-9039-c3a7759481a9") : secret "image-registry-tls" not found
Sep 17 17:36:26.374606 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:26.374576    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-hx6xd" event={"ID":"4893c222-3bba-4071-ae44-3f04bd4c523c","Type":"ContainerStarted","Data":"2c8bde4322362afd8ec7a9220ef8a827a0aa606f5da7aa1dafc86dcf15d178fd"}
Sep 17 17:36:26.395142 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:26.395097    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-hx6xd" podStartSLOduration=2.486798609 podStartE2EDuration="3.395082844s" totalImagesPullingTime="908.284235ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:26.394891418 +0000 UTC m=+97.486198194" watchObservedRunningTime="2026-09-17 17:36:26.395082844 +0000 UTC m=+97.486389608"
Sep 17 17:36:26.870862 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:26.870831    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-registry-tls\") pod \"image-registry-7bd7db6998-stqrm\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") " pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:36:26.871006 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:26.870982    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:36:26.871006 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:26.870996    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bd7db6998-stqrm: secret "image-registry-tls" not found
Sep 17 17:36:26.871075 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:26.871050    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-registry-tls podName:ff45a922-492d-4d9b-9039-c3a7759481a9 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:28.871033345 +0000 UTC m=+99.962340107 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-registry-tls") pod "image-registry-7bd7db6998-stqrm" (UID: "ff45a922-492d-4d9b-9039-c3a7759481a9") : secret "image-registry-tls" not found
Sep 17 17:36:26.892195 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:26.892165    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ntb24"]
Sep 17 17:36:26.893868 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:26.893853    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ntb24"
Sep 17 17:36:26.896264 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:26.896240    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 17:36:26.896381 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:26.896260    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 17:36:26.897012 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:26.896992    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:36:26.897122 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:26.896996    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 17:36:26.897122 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:26.897058    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-7g97q\""
Sep 17 17:36:26.905012 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:26.904993    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ntb24"]
Sep 17 17:36:26.971572 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:26.971545    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n5vk\" (UniqueName: \"kubernetes.io/projected/38abcd71-ad24-4ef8-b192-6124f30e33fd-kube-api-access-4n5vk\") pod \"kube-storage-version-migrator-operator-55889f7b47-ntb24\" (UID: \"38abcd71-ad24-4ef8-b192-6124f30e33fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ntb24"
Sep 17 17:36:26.971697 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:26.971576    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38abcd71-ad24-4ef8-b192-6124f30e33fd-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-ntb24\" (UID: \"38abcd71-ad24-4ef8-b192-6124f30e33fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ntb24"
Sep 17 17:36:26.971697 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:26.971596    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/38abcd71-ad24-4ef8-b192-6124f30e33fd-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-ntb24\" (UID: \"38abcd71-ad24-4ef8-b192-6124f30e33fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ntb24"
Sep 17 17:36:27.072327 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:27.072299    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n5vk\" (UniqueName: \"kubernetes.io/projected/38abcd71-ad24-4ef8-b192-6124f30e33fd-kube-api-access-4n5vk\") pod \"kube-storage-version-migrator-operator-55889f7b47-ntb24\" (UID: \"38abcd71-ad24-4ef8-b192-6124f30e33fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ntb24"
Sep 17 17:36:27.072440 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:27.072332    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38abcd71-ad24-4ef8-b192-6124f30e33fd-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-ntb24\" (UID: \"38abcd71-ad24-4ef8-b192-6124f30e33fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ntb24"
Sep 17 17:36:27.072440 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:27.072355    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/38abcd71-ad24-4ef8-b192-6124f30e33fd-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-ntb24\" (UID: \"38abcd71-ad24-4ef8-b192-6124f30e33fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ntb24"
Sep 17 17:36:27.072967 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:27.072940    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/38abcd71-ad24-4ef8-b192-6124f30e33fd-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-ntb24\" (UID: \"38abcd71-ad24-4ef8-b192-6124f30e33fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ntb24"
Sep 17 17:36:27.074441 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:27.074422    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38abcd71-ad24-4ef8-b192-6124f30e33fd-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-ntb24\" (UID: \"38abcd71-ad24-4ef8-b192-6124f30e33fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ntb24"
Sep 17 17:36:27.085139 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:27.085116    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n5vk\" (UniqueName: \"kubernetes.io/projected/38abcd71-ad24-4ef8-b192-6124f30e33fd-kube-api-access-4n5vk\") pod \"kube-storage-version-migrator-operator-55889f7b47-ntb24\" (UID: \"38abcd71-ad24-4ef8-b192-6124f30e33fd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ntb24"
Sep 17 17:36:27.201940 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:27.201869    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ntb24"
Sep 17 17:36:27.469865 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:27.469792    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ntb24"]
Sep 17 17:36:27.473419 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:36:27.473392    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38abcd71_ad24_4ef8_b192_6124f30e33fd.slice/crio-c140459f662c673e9b02d40a6a67b4ed9f9852ab246d6f9e32a90c06352d541c WatchSource:0}: Error finding container c140459f662c673e9b02d40a6a67b4ed9f9852ab246d6f9e32a90c06352d541c: Status 404 returned error can't find the container with id c140459f662c673e9b02d40a6a67b4ed9f9852ab246d6f9e32a90c06352d541c
Sep 17 17:36:27.676588 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:27.676545    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f11e6c11-e031-485d-8ffa-f71b00f75347-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-5ftqq\" (UID: \"f11e6c11-e031-485d-8ffa-f71b00f75347\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ftqq"
Sep 17 17:36:27.676792 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:27.676699    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:27.676792 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:27.676767    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f11e6c11-e031-485d-8ffa-f71b00f75347-cluster-monitoring-operator-tls podName:f11e6c11-e031-485d-8ffa-f71b00f75347 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:31.676748569 +0000 UTC m=+102.768055317 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f11e6c11-e031-485d-8ffa-f71b00f75347-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-5ftqq" (UID: "f11e6c11-e031-485d-8ffa-f71b00f75347") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:27.844232 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:27.844150    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xvk2c"]
Sep 17 17:36:27.846914 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:27.846893    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xvk2c"
Sep 17 17:36:27.849440 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:27.849418    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 17:36:27.849617 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:27.849443    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 17:36:27.849617 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:27.849532    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 17:36:27.850342 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:27.850328    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 17:36:27.850419 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:27.850360    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4ktmh\""
Sep 17 17:36:27.860416 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:27.860391    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xvk2c"]
Sep 17 17:36:27.878794 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:27.878771    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61cae3db-f72a-4441-bc97-46cc985a16b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvk2c\" (UID: \"61cae3db-f72a-4441-bc97-46cc985a16b2\") " pod="openshift-insights/insights-runtime-extractor-xvk2c"
Sep 17 17:36:27.878899 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:27.878816    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61cae3db-f72a-4441-bc97-46cc985a16b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvk2c\" (UID: \"61cae3db-f72a-4441-bc97-46cc985a16b2\") " pod="openshift-insights/insights-runtime-extractor-xvk2c"
Sep 17 17:36:27.878958 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:27.878909    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/61cae3db-f72a-4441-bc97-46cc985a16b2-crio-socket\") pod \"insights-runtime-extractor-xvk2c\" (UID: \"61cae3db-f72a-4441-bc97-46cc985a16b2\") " pod="openshift-insights/insights-runtime-extractor-xvk2c"
Sep 17 17:36:27.878958 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:27.878942    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/61cae3db-f72a-4441-bc97-46cc985a16b2-data-volume\") pod \"insights-runtime-extractor-xvk2c\" (UID: \"61cae3db-f72a-4441-bc97-46cc985a16b2\") " pod="openshift-insights/insights-runtime-extractor-xvk2c"
Sep 17 17:36:27.879041 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:27.878968    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g45c\" (UniqueName: \"kubernetes.io/projected/61cae3db-f72a-4441-bc97-46cc985a16b2-kube-api-access-5g45c\") pod \"insights-runtime-extractor-xvk2c\" (UID: \"61cae3db-f72a-4441-bc97-46cc985a16b2\") " pod="openshift-insights/insights-runtime-extractor-xvk2c"
Sep 17 17:36:27.980169 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:27.980138    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/61cae3db-f72a-4441-bc97-46cc985a16b2-crio-socket\") pod \"insights-runtime-extractor-xvk2c\" (UID: \"61cae3db-f72a-4441-bc97-46cc985a16b2\") " pod="openshift-insights/insights-runtime-extractor-xvk2c"
Sep 17 17:36:27.980169 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:27.980169    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/61cae3db-f72a-4441-bc97-46cc985a16b2-data-volume\") pod \"insights-runtime-extractor-xvk2c\" (UID: \"61cae3db-f72a-4441-bc97-46cc985a16b2\") " pod="openshift-insights/insights-runtime-extractor-xvk2c"
Sep 17 17:36:27.980348 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:27.980185    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g45c\" (UniqueName: \"kubernetes.io/projected/61cae3db-f72a-4441-bc97-46cc985a16b2-kube-api-access-5g45c\") pod \"insights-runtime-extractor-xvk2c\" (UID: \"61cae3db-f72a-4441-bc97-46cc985a16b2\") " pod="openshift-insights/insights-runtime-extractor-xvk2c"
Sep 17 17:36:27.980348 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:27.980255    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/61cae3db-f72a-4441-bc97-46cc985a16b2-crio-socket\") pod \"insights-runtime-extractor-xvk2c\" (UID: \"61cae3db-f72a-4441-bc97-46cc985a16b2\") " pod="openshift-insights/insights-runtime-extractor-xvk2c"
Sep 17 17:36:27.980348 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:27.980341    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61cae3db-f72a-4441-bc97-46cc985a16b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvk2c\" (UID: \"61cae3db-f72a-4441-bc97-46cc985a16b2\") " pod="openshift-insights/insights-runtime-extractor-xvk2c"
Sep 17 17:36:27.980499 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:27.980373    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61cae3db-f72a-4441-bc97-46cc985a16b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvk2c\" (UID: \"61cae3db-f72a-4441-bc97-46cc985a16b2\") " pod="openshift-insights/insights-runtime-extractor-xvk2c"
Sep 17 17:36:27.980499 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:27.980462    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:36:27.980593 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:27.980501    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:27.980593 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:27.980520    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/61cae3db-f72a-4441-bc97-46cc985a16b2-kube-rbac-proxy-cm podName:61cae3db-f72a-4441-bc97-46cc985a16b2 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:28.480505405 +0000 UTC m=+99.571812152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/61cae3db-f72a-4441-bc97-46cc985a16b2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xvk2c" (UID: "61cae3db-f72a-4441-bc97-46cc985a16b2") : configmap "kube-rbac-proxy" not found
Sep 17 17:36:27.980593 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:27.980464    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/61cae3db-f72a-4441-bc97-46cc985a16b2-data-volume\") pod \"insights-runtime-extractor-xvk2c\" (UID: \"61cae3db-f72a-4441-bc97-46cc985a16b2\") " pod="openshift-insights/insights-runtime-extractor-xvk2c"
Sep 17 17:36:27.980593 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:27.980551    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61cae3db-f72a-4441-bc97-46cc985a16b2-insights-runtime-extractor-tls podName:61cae3db-f72a-4441-bc97-46cc985a16b2 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:28.480535617 +0000 UTC m=+99.571842375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/61cae3db-f72a-4441-bc97-46cc985a16b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xvk2c" (UID: "61cae3db-f72a-4441-bc97-46cc985a16b2") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:27.990068 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:27.990046    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g45c\" (UniqueName: \"kubernetes.io/projected/61cae3db-f72a-4441-bc97-46cc985a16b2-kube-api-access-5g45c\") pod \"insights-runtime-extractor-xvk2c\" (UID: \"61cae3db-f72a-4441-bc97-46cc985a16b2\") " pod="openshift-insights/insights-runtime-extractor-xvk2c"
Sep 17 17:36:28.381019 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:28.380980    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ntb24" event={"ID":"38abcd71-ad24-4ef8-b192-6124f30e33fd","Type":"ContainerStarted","Data":"c140459f662c673e9b02d40a6a67b4ed9f9852ab246d6f9e32a90c06352d541c"}
Sep 17 17:36:28.484338 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:28.484297    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61cae3db-f72a-4441-bc97-46cc985a16b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvk2c\" (UID: \"61cae3db-f72a-4441-bc97-46cc985a16b2\") " pod="openshift-insights/insights-runtime-extractor-xvk2c"
Sep 17 17:36:28.484338 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:28.484337    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61cae3db-f72a-4441-bc97-46cc985a16b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvk2c\" (UID: \"61cae3db-f72a-4441-bc97-46cc985a16b2\") " pod="openshift-insights/insights-runtime-extractor-xvk2c"
Sep 17 17:36:28.484818 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:28.484453    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:28.484818 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:28.484459    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:36:28.484818 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:28.484504    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61cae3db-f72a-4441-bc97-46cc985a16b2-insights-runtime-extractor-tls podName:61cae3db-f72a-4441-bc97-46cc985a16b2 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:29.484491625 +0000 UTC m=+100.575798368 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/61cae3db-f72a-4441-bc97-46cc985a16b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xvk2c" (UID: "61cae3db-f72a-4441-bc97-46cc985a16b2") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:28.484818 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:28.484530    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/61cae3db-f72a-4441-bc97-46cc985a16b2-kube-rbac-proxy-cm podName:61cae3db-f72a-4441-bc97-46cc985a16b2 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:29.4845108 +0000 UTC m=+100.575817564 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/61cae3db-f72a-4441-bc97-46cc985a16b2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xvk2c" (UID: "61cae3db-f72a-4441-bc97-46cc985a16b2") : configmap "kube-rbac-proxy" not found
Sep 17 17:36:28.785691 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:28.785598    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-28g6n"]
Sep 17 17:36:28.787755 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:28.787730    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-28g6n"
Sep 17 17:36:28.790252 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:28.790226    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 17 17:36:28.790379 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:28.790226    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-v5jzp\""
Sep 17 17:36:28.791017 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:28.790992    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 17 17:36:28.791148 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:28.791026    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 17 17:36:28.791148 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:28.791087    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:36:28.797901 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:28.797878    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-28g6n"]
Sep 17 17:36:28.887671 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:28.887639    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/70fb0a3b-a013-4441-9403-d2905c25d596-config\") pod \"service-ca-operator-57cdc4b768-28g6n\" (UID: \"70fb0a3b-a013-4441-9403-d2905c25d596\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-28g6n"
Sep 17 17:36:28.887835 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:28.887695    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-registry-tls\") pod \"image-registry-7bd7db6998-stqrm\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") " pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:36:28.887835 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:28.887790    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/70fb0a3b-a013-4441-9403-d2905c25d596-serving-cert\") pod \"service-ca-operator-57cdc4b768-28g6n\" (UID: \"70fb0a3b-a013-4441-9403-d2905c25d596\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-28g6n"
Sep 17 17:36:28.887835 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:28.887817    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:36:28.887835 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:28.887833    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bd7db6998-stqrm: secret "image-registry-tls" not found
Sep 17 17:36:28.888052 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:28.887857    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzvjf\" (UniqueName: \"kubernetes.io/projected/70fb0a3b-a013-4441-9403-d2905c25d596-kube-api-access-hzvjf\") pod \"service-ca-operator-57cdc4b768-28g6n\" (UID: \"70fb0a3b-a013-4441-9403-d2905c25d596\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-28g6n"
Sep 17 17:36:28.888052 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:28.887961    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-registry-tls podName:ff45a922-492d-4d9b-9039-c3a7759481a9 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:32.887939123 +0000 UTC m=+103.979245875 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-registry-tls") pod "image-registry-7bd7db6998-stqrm" (UID: "ff45a922-492d-4d9b-9039-c3a7759481a9") : secret "image-registry-tls" not found
Sep 17 17:36:28.988630 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:28.988591    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/70fb0a3b-a013-4441-9403-d2905c25d596-config\") pod \"service-ca-operator-57cdc4b768-28g6n\" (UID: \"70fb0a3b-a013-4441-9403-d2905c25d596\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-28g6n"
Sep 17 17:36:28.988826 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:28.988671    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/70fb0a3b-a013-4441-9403-d2905c25d596-serving-cert\") pod \"service-ca-operator-57cdc4b768-28g6n\" (UID: \"70fb0a3b-a013-4441-9403-d2905c25d596\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-28g6n"
Sep 17 17:36:28.988826 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:28.988697    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzvjf\" (UniqueName: \"kubernetes.io/projected/70fb0a3b-a013-4441-9403-d2905c25d596-kube-api-access-hzvjf\") pod \"service-ca-operator-57cdc4b768-28g6n\" (UID: \"70fb0a3b-a013-4441-9403-d2905c25d596\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-28g6n"
Sep 17 17:36:28.989252 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:28.989227    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/70fb0a3b-a013-4441-9403-d2905c25d596-config\") pod \"service-ca-operator-57cdc4b768-28g6n\" (UID: \"70fb0a3b-a013-4441-9403-d2905c25d596\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-28g6n"
Sep 17 17:36:28.991197 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:28.991171    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/70fb0a3b-a013-4441-9403-d2905c25d596-serving-cert\") pod \"service-ca-operator-57cdc4b768-28g6n\" (UID: \"70fb0a3b-a013-4441-9403-d2905c25d596\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-28g6n"
Sep 17 17:36:29.006129 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:29.006105    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzvjf\" (UniqueName: \"kubernetes.io/projected/70fb0a3b-a013-4441-9403-d2905c25d596-kube-api-access-hzvjf\") pod \"service-ca-operator-57cdc4b768-28g6n\" (UID: \"70fb0a3b-a013-4441-9403-d2905c25d596\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-28g6n"
Sep 17 17:36:29.098995 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:29.098964    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-28g6n"
Sep 17 17:36:29.191344 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:29.191301    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dad536d7-d27e-4d63-96e9-a6654126532d-metrics-tls\") pod \"dns-default-d6z4r\" (UID: \"dad536d7-d27e-4d63-96e9-a6654126532d\") " pod="openshift-dns/dns-default-d6z4r"
Sep 17 17:36:29.191527 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:29.191461    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:36:29.191595 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:29.191531    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dad536d7-d27e-4d63-96e9-a6654126532d-metrics-tls podName:dad536d7-d27e-4d63-96e9-a6654126532d nodeName:}" failed. No retries permitted until 2026-09-17 17:37:33.191510245 +0000 UTC m=+164.282817002 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dad536d7-d27e-4d63-96e9-a6654126532d-metrics-tls") pod "dns-default-d6z4r" (UID: "dad536d7-d27e-4d63-96e9-a6654126532d") : secret "dns-default-metrics-tls" not found
Sep 17 17:36:29.385251 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:29.385214    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ntb24" event={"ID":"38abcd71-ad24-4ef8-b192-6124f30e33fd","Type":"ContainerStarted","Data":"493e5fa99c4c4448f0dcc9de5b139333d47db943668ba55ceba39815c033dcba"}
Sep 17 17:36:29.407113 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:29.407059    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ntb24" podStartSLOduration=1.5798894049999999 podStartE2EDuration="3.40703955s" totalImagesPullingTime="1.827150145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:29.405627834 +0000 UTC m=+100.496934623" watchObservedRunningTime="2026-09-17 17:36:29.40703955 +0000 UTC m=+100.498346318"
Sep 17 17:36:29.494583 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:29.494542    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61cae3db-f72a-4441-bc97-46cc985a16b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvk2c\" (UID: \"61cae3db-f72a-4441-bc97-46cc985a16b2\") " pod="openshift-insights/insights-runtime-extractor-xvk2c"
Sep 17 17:36:29.494583 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:29.494579    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61cae3db-f72a-4441-bc97-46cc985a16b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvk2c\" (UID: \"61cae3db-f72a-4441-bc97-46cc985a16b2\") " pod="openshift-insights/insights-runtime-extractor-xvk2c"
Sep 17 17:36:29.495049 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:29.494684    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:36:29.495049 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:29.494757    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/61cae3db-f72a-4441-bc97-46cc985a16b2-kube-rbac-proxy-cm podName:61cae3db-f72a-4441-bc97-46cc985a16b2 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:31.494737617 +0000 UTC m=+102.586044379 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/61cae3db-f72a-4441-bc97-46cc985a16b2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xvk2c" (UID: "61cae3db-f72a-4441-bc97-46cc985a16b2") : configmap "kube-rbac-proxy" not found
Sep 17 17:36:29.495049 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:29.494759    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:29.495049 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:29.494840    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61cae3db-f72a-4441-bc97-46cc985a16b2-insights-runtime-extractor-tls podName:61cae3db-f72a-4441-bc97-46cc985a16b2 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:31.494819062 +0000 UTC m=+102.586125823 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/61cae3db-f72a-4441-bc97-46cc985a16b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xvk2c" (UID: "61cae3db-f72a-4441-bc97-46cc985a16b2") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:29.533744 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:29.533715    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-28g6n"]
Sep 17 17:36:29.536938 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:36:29.536917    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70fb0a3b_a013_4441_9403_d2905c25d596.slice/crio-28cffe83c01a67ee2275f63dc59f4df7f2677465304fa462325eb78d02717471 WatchSource:0}: Error finding container 28cffe83c01a67ee2275f63dc59f4df7f2677465304fa462325eb78d02717471: Status 404 returned error can't find the container with id 28cffe83c01a67ee2275f63dc59f4df7f2677465304fa462325eb78d02717471
Sep 17 17:36:29.999116 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:29.999089    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gzt58_b92cf805-a323-422f-b021-6e8d9bce0099/dns-node-resolver/0.log"
Sep 17 17:36:30.389301 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:30.389256    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-28g6n" event={"ID":"70fb0a3b-a013-4441-9403-d2905c25d596","Type":"ContainerStarted","Data":"28cffe83c01a67ee2275f63dc59f4df7f2677465304fa462325eb78d02717471"}
Sep 17 17:36:31.003422 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:31.003393    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ph456_a20d51c6-5fef-4251-9e8f-ee4318f97741/node-ca/0.log"
Sep 17 17:36:31.394345 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:31.394299    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-28g6n" event={"ID":"70fb0a3b-a013-4441-9403-d2905c25d596","Type":"ContainerStarted","Data":"7682bd9c1239bed3cd2c11020d5a16fed95b318eddf78861206a6c422420d964"}
Sep 17 17:36:31.411533 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:31.411488    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-28g6n" podStartSLOduration=1.827494095 podStartE2EDuration="3.411467887s" totalImagesPullingTime="1.583973792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:31.410703784 +0000 UTC m=+102.502010550" watchObservedRunningTime="2026-09-17 17:36:31.411467887 +0000 UTC m=+102.502774652"
Sep 17 17:36:31.513728 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:31.513685    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61cae3db-f72a-4441-bc97-46cc985a16b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvk2c\" (UID: \"61cae3db-f72a-4441-bc97-46cc985a16b2\") " pod="openshift-insights/insights-runtime-extractor-xvk2c"
Sep 17 17:36:31.513728 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:31.513732    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61cae3db-f72a-4441-bc97-46cc985a16b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvk2c\" (UID: \"61cae3db-f72a-4441-bc97-46cc985a16b2\") " pod="openshift-insights/insights-runtime-extractor-xvk2c"
Sep 17 17:36:31.513983 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:31.513799    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:36:31.513983 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:31.513822    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:31.513983 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:31.513881    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/61cae3db-f72a-4441-bc97-46cc985a16b2-kube-rbac-proxy-cm podName:61cae3db-f72a-4441-bc97-46cc985a16b2 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:35.513862453 +0000 UTC m=+106.605169197 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/61cae3db-f72a-4441-bc97-46cc985a16b2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xvk2c" (UID: "61cae3db-f72a-4441-bc97-46cc985a16b2") : configmap "kube-rbac-proxy" not found
Sep 17 17:36:31.513983 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:31.513900    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61cae3db-f72a-4441-bc97-46cc985a16b2-insights-runtime-extractor-tls podName:61cae3db-f72a-4441-bc97-46cc985a16b2 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:35.513892143 +0000 UTC m=+106.605198886 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/61cae3db-f72a-4441-bc97-46cc985a16b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xvk2c" (UID: "61cae3db-f72a-4441-bc97-46cc985a16b2") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:31.715869 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:31.715775    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f11e6c11-e031-485d-8ffa-f71b00f75347-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-5ftqq\" (UID: \"f11e6c11-e031-485d-8ffa-f71b00f75347\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ftqq"
Sep 17 17:36:31.716012 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:31.715943    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:31.716055 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:31.716028    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f11e6c11-e031-485d-8ffa-f71b00f75347-cluster-monitoring-operator-tls podName:f11e6c11-e031-485d-8ffa-f71b00f75347 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:39.716005337 +0000 UTC m=+110.807312086 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f11e6c11-e031-485d-8ffa-f71b00f75347-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-5ftqq" (UID: "f11e6c11-e031-485d-8ffa-f71b00f75347") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:31.799052 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:31.799017    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-q5nq9_be8c6ba8-3ad9-4988-b871-01c0d8807b10/serve-healthcheck-canary/0.log"
Sep 17 17:36:32.925779 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:32.925745    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-registry-tls\") pod \"image-registry-7bd7db6998-stqrm\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") " pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:36:32.926186 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:32.925902    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:36:32.926186 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:32.925918    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bd7db6998-stqrm: secret "image-registry-tls" not found
Sep 17 17:36:32.926186 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:32.925973    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-registry-tls podName:ff45a922-492d-4d9b-9039-c3a7759481a9 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:40.925960674 +0000 UTC m=+112.017267417 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-registry-tls") pod "image-registry-7bd7db6998-stqrm" (UID: "ff45a922-492d-4d9b-9039-c3a7759481a9") : secret "image-registry-tls" not found
Sep 17 17:36:33.322614 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:33.322544    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ksb2q"
Sep 17 17:36:35.546791 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:35.546760    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61cae3db-f72a-4441-bc97-46cc985a16b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvk2c\" (UID: \"61cae3db-f72a-4441-bc97-46cc985a16b2\") " pod="openshift-insights/insights-runtime-extractor-xvk2c"
Sep 17 17:36:35.546791 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:35.546791    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61cae3db-f72a-4441-bc97-46cc985a16b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvk2c\" (UID: \"61cae3db-f72a-4441-bc97-46cc985a16b2\") " pod="openshift-insights/insights-runtime-extractor-xvk2c"
Sep 17 17:36:35.547287 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:35.546885    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:36:35.547287 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:35.546954    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/61cae3db-f72a-4441-bc97-46cc985a16b2-kube-rbac-proxy-cm podName:61cae3db-f72a-4441-bc97-46cc985a16b2 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:43.5469354 +0000 UTC m=+114.638242171 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/61cae3db-f72a-4441-bc97-46cc985a16b2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xvk2c" (UID: "61cae3db-f72a-4441-bc97-46cc985a16b2") : configmap "kube-rbac-proxy" not found
Sep 17 17:36:35.547287 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:35.546969    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:35.547287 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:35.547004    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61cae3db-f72a-4441-bc97-46cc985a16b2-insights-runtime-extractor-tls podName:61cae3db-f72a-4441-bc97-46cc985a16b2 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:43.546994014 +0000 UTC m=+114.638300764 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/61cae3db-f72a-4441-bc97-46cc985a16b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xvk2c" (UID: "61cae3db-f72a-4441-bc97-46cc985a16b2") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:39.779470 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:39.779433    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f11e6c11-e031-485d-8ffa-f71b00f75347-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-5ftqq\" (UID: \"f11e6c11-e031-485d-8ffa-f71b00f75347\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ftqq"
Sep 17 17:36:39.779860 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:39.779571    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:39.779860 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:39.779642    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f11e6c11-e031-485d-8ffa-f71b00f75347-cluster-monitoring-operator-tls podName:f11e6c11-e031-485d-8ffa-f71b00f75347 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:55.779624822 +0000 UTC m=+126.870931565 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f11e6c11-e031-485d-8ffa-f71b00f75347-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-5ftqq" (UID: "f11e6c11-e031-485d-8ffa-f71b00f75347") : secret "cluster-monitoring-operator-tls" not found
Sep 17 17:36:40.988388 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:40.988356    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-registry-tls\") pod \"image-registry-7bd7db6998-stqrm\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") " pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:36:40.988783 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:40.988469    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:36:40.988783 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:40.988482    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bd7db6998-stqrm: secret "image-registry-tls" not found
Sep 17 17:36:40.988783 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:40.988528    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-registry-tls podName:ff45a922-492d-4d9b-9039-c3a7759481a9 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:56.988516292 +0000 UTC m=+128.079823035 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-registry-tls") pod "image-registry-7bd7db6998-stqrm" (UID: "ff45a922-492d-4d9b-9039-c3a7759481a9") : secret "image-registry-tls" not found
Sep 17 17:36:43.609339 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:43.609305    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61cae3db-f72a-4441-bc97-46cc985a16b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvk2c\" (UID: \"61cae3db-f72a-4441-bc97-46cc985a16b2\") " pod="openshift-insights/insights-runtime-extractor-xvk2c"
Sep 17 17:36:43.609339 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:43.609340    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61cae3db-f72a-4441-bc97-46cc985a16b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvk2c\" (UID: \"61cae3db-f72a-4441-bc97-46cc985a16b2\") " pod="openshift-insights/insights-runtime-extractor-xvk2c"
Sep 17 17:36:43.609902 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:43.609473    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:43.609902 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:36:43.609520    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61cae3db-f72a-4441-bc97-46cc985a16b2-insights-runtime-extractor-tls podName:61cae3db-f72a-4441-bc97-46cc985a16b2 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:59.60950555 +0000 UTC m=+130.700812293 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/61cae3db-f72a-4441-bc97-46cc985a16b2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xvk2c" (UID: "61cae3db-f72a-4441-bc97-46cc985a16b2") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:36:43.610016 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:43.609965    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61cae3db-f72a-4441-bc97-46cc985a16b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xvk2c\" (UID: \"61cae3db-f72a-4441-bc97-46cc985a16b2\") " pod="openshift-insights/insights-runtime-extractor-xvk2c"
Sep 17 17:36:55.799417 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:55.799383    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f11e6c11-e031-485d-8ffa-f71b00f75347-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-5ftqq\" (UID: \"f11e6c11-e031-485d-8ffa-f71b00f75347\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ftqq"
Sep 17 17:36:55.801838 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:55.801813    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f11e6c11-e031-485d-8ffa-f71b00f75347-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-5ftqq\" (UID: \"f11e6c11-e031-485d-8ffa-f71b00f75347\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ftqq"
Sep 17 17:36:56.065745 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:56.065654    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-268t5\""
Sep 17 17:36:56.073705 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:56.073686    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ftqq"
Sep 17 17:36:56.344468 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:56.344393    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ftqq"]
Sep 17 17:36:56.347401 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:36:56.347364    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf11e6c11_e031_485d_8ffa_f71b00f75347.slice/crio-bac319866c8124a429d07f279819f5c6b4a1d2c33e885a41ecd585ac6f77bb92 WatchSource:0}: Error finding container bac319866c8124a429d07f279819f5c6b4a1d2c33e885a41ecd585ac6f77bb92: Status 404 returned error can't find the container with id bac319866c8124a429d07f279819f5c6b4a1d2c33e885a41ecd585ac6f77bb92
Sep 17 17:36:56.464760 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:56.464727    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ftqq" event={"ID":"f11e6c11-e031-485d-8ffa-f71b00f75347","Type":"ContainerStarted","Data":"bac319866c8124a429d07f279819f5c6b4a1d2c33e885a41ecd585ac6f77bb92"}
Sep 17 17:36:57.008236 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:57.008200    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-registry-tls\") pod \"image-registry-7bd7db6998-stqrm\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") " pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:36:57.010488 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:57.010471    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-registry-tls\") pod \"image-registry-7bd7db6998-stqrm\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") " pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:36:57.039645 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:57.039618    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-55z8g\""
Sep 17 17:36:57.047812 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:57.047792    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:36:57.328605 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:57.328418    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bd7db6998-stqrm"]
Sep 17 17:36:57.330832 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:36:57.330789    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff45a922_492d_4d9b_9039_c3a7759481a9.slice/crio-500d3a738ed9a5bfaf5159291ef713dad8622d4f47aca63e45bd98058a8de912 WatchSource:0}: Error finding container 500d3a738ed9a5bfaf5159291ef713dad8622d4f47aca63e45bd98058a8de912: Status 404 returned error can't find the container with id 500d3a738ed9a5bfaf5159291ef713dad8622d4f47aca63e45bd98058a8de912
Sep 17 17:36:57.471333 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:57.471297    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bd7db6998-stqrm" event={"ID":"ff45a922-492d-4d9b-9039-c3a7759481a9","Type":"ContainerStarted","Data":"c095a4f9a92da0134681f03ffe34256be370a74239305cc820501f8afb6b5432"}
Sep 17 17:36:57.471496 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:57.471360    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bd7db6998-stqrm" event={"ID":"ff45a922-492d-4d9b-9039-c3a7759481a9","Type":"ContainerStarted","Data":"500d3a738ed9a5bfaf5159291ef713dad8622d4f47aca63e45bd98058a8de912"}
Sep 17 17:36:57.471618 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:57.471592    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:36:57.493870 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:57.493813    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7bd7db6998-stqrm" podStartSLOduration=32.493790143 podStartE2EDuration="32.493790143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:57.492986846 +0000 UTC m=+128.584293636" watchObservedRunningTime="2026-09-17 17:36:57.493790143 +0000 UTC m=+128.585096905"
Sep 17 17:36:58.475786 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:58.475694    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ftqq" event={"ID":"f11e6c11-e031-485d-8ffa-f71b00f75347","Type":"ContainerStarted","Data":"aa394f017c2cd15f228780eba9f9e04dd4b60483cca7df1869ba6578875242e9"}
Sep 17 17:36:58.512628 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:58.512582    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-5ftqq" podStartSLOduration=33.64218713 podStartE2EDuration="35.51256289s" totalImagesPullingTime="1.87037576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:36:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:58.512010242 +0000 UTC m=+129.603317030" watchObservedRunningTime="2026-09-17 17:36:58.51256289 +0000 UTC m=+129.603869655"
Sep 17 17:36:59.630928 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:59.630891    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61cae3db-f72a-4441-bc97-46cc985a16b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvk2c\" (UID: \"61cae3db-f72a-4441-bc97-46cc985a16b2\") " pod="openshift-insights/insights-runtime-extractor-xvk2c"
Sep 17 17:36:59.633192 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:59.633168    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61cae3db-f72a-4441-bc97-46cc985a16b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xvk2c\" (UID: \"61cae3db-f72a-4441-bc97-46cc985a16b2\") " pod="openshift-insights/insights-runtime-extractor-xvk2c"
Sep 17 17:36:59.658948 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:59.658926    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4ktmh\""
Sep 17 17:36:59.667244 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:59.667217    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xvk2c"
Sep 17 17:36:59.935101 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:36:59.935001    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xvk2c"]
Sep 17 17:36:59.937482 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:36:59.937449    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61cae3db_f72a_4441_bc97_46cc985a16b2.slice/crio-cfcb8642d21a9f8fb477b488d81b521689a65b031d6232b45330c3bef6cc08ea WatchSource:0}: Error finding container cfcb8642d21a9f8fb477b488d81b521689a65b031d6232b45330c3bef6cc08ea: Status 404 returned error can't find the container with id cfcb8642d21a9f8fb477b488d81b521689a65b031d6232b45330c3bef6cc08ea
Sep 17 17:37:00.487847 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:00.487809    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xvk2c" event={"ID":"61cae3db-f72a-4441-bc97-46cc985a16b2","Type":"ContainerStarted","Data":"778b81006dca78b9edfc3886fe6d770d907a420bb5fbb0038ff12c2f58f78c2c"}
Sep 17 17:37:00.488017 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:00.487898    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xvk2c" event={"ID":"61cae3db-f72a-4441-bc97-46cc985a16b2","Type":"ContainerStarted","Data":"cfcb8642d21a9f8fb477b488d81b521689a65b031d6232b45330c3bef6cc08ea"}
Sep 17 17:37:00.739735 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:00.739592    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae10190f-448c-4e69-a268-85f77c85d7b6-metrics-certs\") pod \"network-metrics-daemon-fh98p\" (UID: \"ae10190f-448c-4e69-a268-85f77c85d7b6\") " pod="openshift-multus/network-metrics-daemon-fh98p"
Sep 17 17:37:00.741923 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:00.741900    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae10190f-448c-4e69-a268-85f77c85d7b6-metrics-certs\") pod \"network-metrics-daemon-fh98p\" (UID: \"ae10190f-448c-4e69-a268-85f77c85d7b6\") " pod="openshift-multus/network-metrics-daemon-fh98p"
Sep 17 17:37:00.846700 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:00.846654    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j84n5\""
Sep 17 17:37:00.854768 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:00.854748    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fh98p"
Sep 17 17:37:01.123082 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:01.123049    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fh98p"]
Sep 17 17:37:01.126027 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:37:01.126000    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae10190f_448c_4e69_a268_85f77c85d7b6.slice/crio-c1442db6bef4f1d625d35d328d580d0f03e0797139838e60725dc54135b9b442 WatchSource:0}: Error finding container c1442db6bef4f1d625d35d328d580d0f03e0797139838e60725dc54135b9b442: Status 404 returned error can't find the container with id c1442db6bef4f1d625d35d328d580d0f03e0797139838e60725dc54135b9b442
Sep 17 17:37:01.493511 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:01.493414    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xvk2c" event={"ID":"61cae3db-f72a-4441-bc97-46cc985a16b2","Type":"ContainerStarted","Data":"b9d90759ff238b04b74c1019f9d416a3b2dd1bd64a23298738261fa7e5a682f1"}
Sep 17 17:37:01.494740 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:01.494716    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fh98p" event={"ID":"ae10190f-448c-4e69-a268-85f77c85d7b6","Type":"ContainerStarted","Data":"c1442db6bef4f1d625d35d328d580d0f03e0797139838e60725dc54135b9b442"}
Sep 17 17:37:02.078355 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:02.078321    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6884b6f86-vmkrk"]
Sep 17 17:37:02.080483 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:02.080427    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6884b6f86-vmkrk"
Sep 17 17:37:02.083886 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:02.083820    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 17:37:02.084578 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:02.084555    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 17:37:02.084715 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:02.084681    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 17:37:02.088824 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:02.088802    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 17:37:02.089246 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:02.089228    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 17:37:02.089343 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:02.089228    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 17:37:02.091698 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:02.091682    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 17:37:02.104590 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:02.104566    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6884b6f86-vmkrk"]
Sep 17 17:37:02.126371 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:02.126339    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7bd7db6998-stqrm"]
Sep 17 17:37:02.251231 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:02.251196    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/29b8a4df-12f1-405c-8205-9f40e69d3d8f-ca\") pod \"cluster-proxy-proxy-agent-6884b6f86-vmkrk\" (UID: \"29b8a4df-12f1-405c-8205-9f40e69d3d8f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6884b6f86-vmkrk"
Sep 17 17:37:02.251391 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:02.251246    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/29b8a4df-12f1-405c-8205-9f40e69d3d8f-hub\") pod \"cluster-proxy-proxy-agent-6884b6f86-vmkrk\" (UID: \"29b8a4df-12f1-405c-8205-9f40e69d3d8f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6884b6f86-vmkrk"
Sep 17 17:37:02.251391 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:02.251268    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dxdj\" (UniqueName: \"kubernetes.io/projected/29b8a4df-12f1-405c-8205-9f40e69d3d8f-kube-api-access-8dxdj\") pod \"cluster-proxy-proxy-agent-6884b6f86-vmkrk\" (UID: \"29b8a4df-12f1-405c-8205-9f40e69d3d8f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6884b6f86-vmkrk"
Sep 17 17:37:02.251391 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:02.251346    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/29b8a4df-12f1-405c-8205-9f40e69d3d8f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6884b6f86-vmkrk\" (UID: \"29b8a4df-12f1-405c-8205-9f40e69d3d8f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6884b6f86-vmkrk"
Sep 17 17:37:02.251391 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:02.251391    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/29b8a4df-12f1-405c-8205-9f40e69d3d8f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6884b6f86-vmkrk\" (UID: \"29b8a4df-12f1-405c-8205-9f40e69d3d8f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6884b6f86-vmkrk"
Sep 17 17:37:02.251565 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:02.251408    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/29b8a4df-12f1-405c-8205-9f40e69d3d8f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6884b6f86-vmkrk\" (UID: \"29b8a4df-12f1-405c-8205-9f40e69d3d8f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6884b6f86-vmkrk"
Sep 17 17:37:02.352527 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:02.352490    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/29b8a4df-12f1-405c-8205-9f40e69d3d8f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6884b6f86-vmkrk\" (UID: \"29b8a4df-12f1-405c-8205-9f40e69d3d8f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6884b6f86-vmkrk"
Sep 17 17:37:02.353170 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:02.352542    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/29b8a4df-12f1-405c-8205-9f40e69d3d8f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6884b6f86-vmkrk\" (UID: \"29b8a4df-12f1-405c-8205-9f40e69d3d8f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6884b6f86-vmkrk"
Sep 17 17:37:02.353170 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:02.352563    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/29b8a4df-12f1-405c-8205-9f40e69d3d8f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6884b6f86-vmkrk\" (UID: \"29b8a4df-12f1-405c-8205-9f40e69d3d8f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6884b6f86-vmkrk"
Sep 17 17:37:02.353170 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:02.352636    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/29b8a4df-12f1-405c-8205-9f40e69d3d8f-ca\") pod \"cluster-proxy-proxy-agent-6884b6f86-vmkrk\" (UID: \"29b8a4df-12f1-405c-8205-9f40e69d3d8f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6884b6f86-vmkrk"
Sep 17 17:37:02.353170 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:02.352687    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/29b8a4df-12f1-405c-8205-9f40e69d3d8f-hub\") pod \"cluster-proxy-proxy-agent-6884b6f86-vmkrk\" (UID: \"29b8a4df-12f1-405c-8205-9f40e69d3d8f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6884b6f86-vmkrk"
Sep 17 17:37:02.353170 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:02.352712    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dxdj\" (UniqueName: \"kubernetes.io/projected/29b8a4df-12f1-405c-8205-9f40e69d3d8f-kube-api-access-8dxdj\") pod \"cluster-proxy-proxy-agent-6884b6f86-vmkrk\" (UID: \"29b8a4df-12f1-405c-8205-9f40e69d3d8f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6884b6f86-vmkrk"
Sep 17 17:37:02.353469 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:02.353296    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/29b8a4df-12f1-405c-8205-9f40e69d3d8f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6884b6f86-vmkrk\" (UID: \"29b8a4df-12f1-405c-8205-9f40e69d3d8f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6884b6f86-vmkrk"
Sep 17 17:37:02.355517 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:02.355483    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/29b8a4df-12f1-405c-8205-9f40e69d3d8f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6884b6f86-vmkrk\" (UID: \"29b8a4df-12f1-405c-8205-9f40e69d3d8f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6884b6f86-vmkrk"
Sep 17 17:37:02.355609 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:02.355525    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/29b8a4df-12f1-405c-8205-9f40e69d3d8f-hub\") pod \"cluster-proxy-proxy-agent-6884b6f86-vmkrk\" (UID: \"29b8a4df-12f1-405c-8205-9f40e69d3d8f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6884b6f86-vmkrk"
Sep 17 17:37:02.355609 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:02.355545    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/29b8a4df-12f1-405c-8205-9f40e69d3d8f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6884b6f86-vmkrk\" (UID: \"29b8a4df-12f1-405c-8205-9f40e69d3d8f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6884b6f86-vmkrk"
Sep 17 17:37:02.355609 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:02.355545    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/29b8a4df-12f1-405c-8205-9f40e69d3d8f-ca\") pod \"cluster-proxy-proxy-agent-6884b6f86-vmkrk\" (UID: \"29b8a4df-12f1-405c-8205-9f40e69d3d8f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6884b6f86-vmkrk"
Sep 17 17:37:02.369617 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:02.369593    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dxdj\" (UniqueName: \"kubernetes.io/projected/29b8a4df-12f1-405c-8205-9f40e69d3d8f-kube-api-access-8dxdj\") pod \"cluster-proxy-proxy-agent-6884b6f86-vmkrk\" (UID: \"29b8a4df-12f1-405c-8205-9f40e69d3d8f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6884b6f86-vmkrk"
Sep 17 17:37:02.400443 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:02.400416    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6884b6f86-vmkrk"
Sep 17 17:37:02.500910 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:02.500881    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xvk2c" event={"ID":"61cae3db-f72a-4441-bc97-46cc985a16b2","Type":"ContainerStarted","Data":"df8afc406322c215dc690bce82ba2c13bb221003ea9874c5e6a50d94d2e58765"}
Sep 17 17:37:02.537367 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:02.537313    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xvk2c" podStartSLOduration=33.757932376 podStartE2EDuration="35.53729266s" totalImagesPullingTime="1.779360284s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:36:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:02.537045129 +0000 UTC m=+133.628351931" watchObservedRunningTime="2026-09-17 17:37:02.53729266 +0000 UTC m=+133.628599425"
Sep 17 17:37:02.683504 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:02.683426    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6884b6f86-vmkrk"]
Sep 17 17:37:02.686923 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:37:02.686890    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29b8a4df_12f1_405c_8205_9f40e69d3d8f.slice/crio-3ea7f7974fb3f327f006817b996e914ed8e47df4f0c30fb580286fabaa4bde0f WatchSource:0}: Error finding container 3ea7f7974fb3f327f006817b996e914ed8e47df4f0c30fb580286fabaa4bde0f: Status 404 returned error can't find the container with id 3ea7f7974fb3f327f006817b996e914ed8e47df4f0c30fb580286fabaa4bde0f
Sep 17 17:37:03.510015 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:03.509972    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fh98p" event={"ID":"ae10190f-448c-4e69-a268-85f77c85d7b6","Type":"ContainerStarted","Data":"c6506cb379e9895cb32694d3f0dcea9ce2bc5f978254695eab2403e2e517a949"}
Sep 17 17:37:03.510483 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:03.510046    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fh98p" event={"ID":"ae10190f-448c-4e69-a268-85f77c85d7b6","Type":"ContainerStarted","Data":"2654c5b67bceedfd2bc745fe8fd23ca818c529a22da11d3ed7fb46a4d61ff54f"}
Sep 17 17:37:03.512136 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:03.512104    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6884b6f86-vmkrk" event={"ID":"29b8a4df-12f1-405c-8205-9f40e69d3d8f","Type":"ContainerStarted","Data":"3ea7f7974fb3f327f006817b996e914ed8e47df4f0c30fb580286fabaa4bde0f"}
Sep 17 17:37:03.543964 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:03.543908    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fh98p" podStartSLOduration=131.264927399 podStartE2EDuration="2m12.543887222s" totalImagesPullingTime="1.278959823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:03.541267896 +0000 UTC m=+134.632574679" watchObservedRunningTime="2026-09-17 17:37:03.543887222 +0000 UTC m=+134.635193989"
Sep 17 17:37:06.525865 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:06.525830    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6884b6f86-vmkrk" event={"ID":"29b8a4df-12f1-405c-8205-9f40e69d3d8f","Type":"ContainerStarted","Data":"34acf5d81024e60ef6874ce35852803af22c26dfd5df849967071b591b1294af"}
Sep 17 17:37:06.526264 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:06.525880    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6884b6f86-vmkrk" event={"ID":"29b8a4df-12f1-405c-8205-9f40e69d3d8f","Type":"ContainerStarted","Data":"1094242f9a93c83d1a0a293e970424fbdea7ba136bacf94c01314c7f42b14d35"}
Sep 17 17:37:06.526264 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:06.525901    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6884b6f86-vmkrk" event={"ID":"29b8a4df-12f1-405c-8205-9f40e69d3d8f","Type":"ContainerStarted","Data":"d375578917193b7db3ab19c47b52b429b724ea1036cdfc730604644f37cff5ef"}
Sep 17 17:37:06.546636 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:06.546596    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6884b6f86-vmkrk" podStartSLOduration=1.1958902550000001 podStartE2EDuration="4.546580325s" totalImagesPullingTime="3.35069007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:06.545469426 +0000 UTC m=+137.636776215" watchObservedRunningTime="2026-09-17 17:37:06.546580325 +0000 UTC m=+137.637887090"
Sep 17 17:37:10.504199 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.504164    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-qn64b"]
Sep 17 17:37:10.506234 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.506218    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-qn64b"
Sep 17 17:37:10.510000 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.509977    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 17:37:10.510125 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.509979    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 17:37:10.510125 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.510002    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 17:37:10.510125 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.510031    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-vfbqz\""
Sep 17 17:37:10.510125 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.510003    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 17:37:10.521212 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.521190    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-qn64b"]
Sep 17 17:37:10.524466 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.524444    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-t5vx9"]
Sep 17 17:37:10.526404 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.526387    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t5vx9"
Sep 17 17:37:10.528783 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.528766    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 17:37:10.528888 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.528835    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 17:37:10.528951 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.528888    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kjs7l\""
Sep 17 17:37:10.529197 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.529181    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 17:37:10.614914 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.614885    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/215cb362-1c54-4086-83a8-99c96e59e90d-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-qn64b\" (UID: \"215cb362-1c54-4086-83a8-99c96e59e90d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qn64b"
Sep 17 17:37:10.614914 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.614916    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/215cb362-1c54-4086-83a8-99c96e59e90d-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-qn64b\" (UID: \"215cb362-1c54-4086-83a8-99c96e59e90d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qn64b"
Sep 17 17:37:10.615126 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.614943    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp87g\" (UniqueName: \"kubernetes.io/projected/215cb362-1c54-4086-83a8-99c96e59e90d-kube-api-access-dp87g\") pod \"kube-state-metrics-84c885658c-qn64b\" (UID: \"215cb362-1c54-4086-83a8-99c96e59e90d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qn64b"
Sep 17 17:37:10.615126 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.615039    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/215cb362-1c54-4086-83a8-99c96e59e90d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-qn64b\" (UID: \"215cb362-1c54-4086-83a8-99c96e59e90d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qn64b"
Sep 17 17:37:10.615126 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.615114    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/215cb362-1c54-4086-83a8-99c96e59e90d-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-qn64b\" (UID: \"215cb362-1c54-4086-83a8-99c96e59e90d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qn64b"
Sep 17 17:37:10.615264 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.615154    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/215cb362-1c54-4086-83a8-99c96e59e90d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-qn64b\" (UID: \"215cb362-1c54-4086-83a8-99c96e59e90d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qn64b"
Sep 17 17:37:10.715719 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.715685    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/031c5c73-85cc-465f-b4c5-ac1b82042ac6-sys\") pod \"node-exporter-t5vx9\" (UID: \"031c5c73-85cc-465f-b4c5-ac1b82042ac6\") " pod="openshift-monitoring/node-exporter-t5vx9"
Sep 17 17:37:10.715719 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.715719    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/031c5c73-85cc-465f-b4c5-ac1b82042ac6-root\") pod \"node-exporter-t5vx9\" (UID: \"031c5c73-85cc-465f-b4c5-ac1b82042ac6\") " pod="openshift-monitoring/node-exporter-t5vx9"
Sep 17 17:37:10.715929 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.715738    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/031c5c73-85cc-465f-b4c5-ac1b82042ac6-node-exporter-textfile\") pod \"node-exporter-t5vx9\" (UID: \"031c5c73-85cc-465f-b4c5-ac1b82042ac6\") " pod="openshift-monitoring/node-exporter-t5vx9"
Sep 17 17:37:10.715929 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.715764    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/031c5c73-85cc-465f-b4c5-ac1b82042ac6-node-exporter-accelerators-collector-config\") pod \"node-exporter-t5vx9\" (UID: \"031c5c73-85cc-465f-b4c5-ac1b82042ac6\") " pod="openshift-monitoring/node-exporter-t5vx9"
Sep 17 17:37:10.715929 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.715818    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/215cb362-1c54-4086-83a8-99c96e59e90d-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-qn64b\" (UID: \"215cb362-1c54-4086-83a8-99c96e59e90d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qn64b"
Sep 17 17:37:10.715929 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.715838    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/215cb362-1c54-4086-83a8-99c96e59e90d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-qn64b\" (UID: \"215cb362-1c54-4086-83a8-99c96e59e90d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qn64b"
Sep 17 17:37:10.715929 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.715855    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/215cb362-1c54-4086-83a8-99c96e59e90d-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-qn64b\" (UID: \"215cb362-1c54-4086-83a8-99c96e59e90d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qn64b"
Sep 17 17:37:10.715929 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.715869    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/031c5c73-85cc-465f-b4c5-ac1b82042ac6-node-exporter-tls\") pod \"node-exporter-t5vx9\" (UID: \"031c5c73-85cc-465f-b4c5-ac1b82042ac6\") " pod="openshift-monitoring/node-exporter-t5vx9"
Sep 17 17:37:10.715929 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.715889    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/215cb362-1c54-4086-83a8-99c96e59e90d-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-qn64b\" (UID: \"215cb362-1c54-4086-83a8-99c96e59e90d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qn64b"
Sep 17 17:37:10.716249 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.715983    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/031c5c73-85cc-465f-b4c5-ac1b82042ac6-metrics-client-ca\") pod \"node-exporter-t5vx9\" (UID: \"031c5c73-85cc-465f-b4c5-ac1b82042ac6\") " pod="openshift-monitoring/node-exporter-t5vx9"
Sep 17 17:37:10.716249 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.716002    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/031c5c73-85cc-465f-b4c5-ac1b82042ac6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t5vx9\" (UID: \"031c5c73-85cc-465f-b4c5-ac1b82042ac6\") " pod="openshift-monitoring/node-exporter-t5vx9"
Sep 17 17:37:10.716249 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.716078    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp87g\" (UniqueName: \"kubernetes.io/projected/215cb362-1c54-4086-83a8-99c96e59e90d-kube-api-access-dp87g\") pod \"kube-state-metrics-84c885658c-qn64b\" (UID: \"215cb362-1c54-4086-83a8-99c96e59e90d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qn64b"
Sep 17 17:37:10.716249 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.716132    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/031c5c73-85cc-465f-b4c5-ac1b82042ac6-node-exporter-wtmp\") pod \"node-exporter-t5vx9\" (UID: \"031c5c73-85cc-465f-b4c5-ac1b82042ac6\") " pod="openshift-monitoring/node-exporter-t5vx9"
Sep 17 17:37:10.716249 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.716243    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxdfx\" (UniqueName: \"kubernetes.io/projected/031c5c73-85cc-465f-b4c5-ac1b82042ac6-kube-api-access-zxdfx\") pod \"node-exporter-t5vx9\" (UID: \"031c5c73-85cc-465f-b4c5-ac1b82042ac6\") " pod="openshift-monitoring/node-exporter-t5vx9"
Sep 17 17:37:10.716471 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.716249    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/215cb362-1c54-4086-83a8-99c96e59e90d-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-qn64b\" (UID: \"215cb362-1c54-4086-83a8-99c96e59e90d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qn64b"
Sep 17 17:37:10.716471 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.716294    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/215cb362-1c54-4086-83a8-99c96e59e90d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-qn64b\" (UID: \"215cb362-1c54-4086-83a8-99c96e59e90d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qn64b"
Sep 17 17:37:10.716601 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.716583    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/215cb362-1c54-4086-83a8-99c96e59e90d-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-qn64b\" (UID: \"215cb362-1c54-4086-83a8-99c96e59e90d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qn64b"
Sep 17 17:37:10.716958 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.716937    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/215cb362-1c54-4086-83a8-99c96e59e90d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-qn64b\" (UID: \"215cb362-1c54-4086-83a8-99c96e59e90d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qn64b"
Sep 17 17:37:10.718482 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.718459    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/215cb362-1c54-4086-83a8-99c96e59e90d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-qn64b\" (UID: \"215cb362-1c54-4086-83a8-99c96e59e90d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qn64b"
Sep 17 17:37:10.718654 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.718631    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/215cb362-1c54-4086-83a8-99c96e59e90d-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-qn64b\" (UID: \"215cb362-1c54-4086-83a8-99c96e59e90d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qn64b"
Sep 17 17:37:10.725278 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.725252    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp87g\" (UniqueName: \"kubernetes.io/projected/215cb362-1c54-4086-83a8-99c96e59e90d-kube-api-access-dp87g\") pod \"kube-state-metrics-84c885658c-qn64b\" (UID: \"215cb362-1c54-4086-83a8-99c96e59e90d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qn64b"
Sep 17 17:37:10.815204 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.815142    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-qn64b"
Sep 17 17:37:10.817432 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.817407    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/031c5c73-85cc-465f-b4c5-ac1b82042ac6-root\") pod \"node-exporter-t5vx9\" (UID: \"031c5c73-85cc-465f-b4c5-ac1b82042ac6\") " pod="openshift-monitoring/node-exporter-t5vx9"
Sep 17 17:37:10.817530 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.817445    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/031c5c73-85cc-465f-b4c5-ac1b82042ac6-node-exporter-textfile\") pod \"node-exporter-t5vx9\" (UID: \"031c5c73-85cc-465f-b4c5-ac1b82042ac6\") " pod="openshift-monitoring/node-exporter-t5vx9"
Sep 17 17:37:10.817530 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.817462    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/031c5c73-85cc-465f-b4c5-ac1b82042ac6-node-exporter-accelerators-collector-config\") pod \"node-exporter-t5vx9\" (UID: \"031c5c73-85cc-465f-b4c5-ac1b82042ac6\") " pod="openshift-monitoring/node-exporter-t5vx9"
Sep 17 17:37:10.817628 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.817521    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/031c5c73-85cc-465f-b4c5-ac1b82042ac6-root\") pod \"node-exporter-t5vx9\" (UID: \"031c5c73-85cc-465f-b4c5-ac1b82042ac6\") " pod="openshift-monitoring/node-exporter-t5vx9"
Sep 17 17:37:10.817628 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.817609    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/031c5c73-85cc-465f-b4c5-ac1b82042ac6-node-exporter-tls\") pod \"node-exporter-t5vx9\" (UID: \"031c5c73-85cc-465f-b4c5-ac1b82042ac6\") " pod="openshift-monitoring/node-exporter-t5vx9"
Sep 17 17:37:10.818146 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.817640    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/031c5c73-85cc-465f-b4c5-ac1b82042ac6-metrics-client-ca\") pod \"node-exporter-t5vx9\" (UID: \"031c5c73-85cc-465f-b4c5-ac1b82042ac6\") " pod="openshift-monitoring/node-exporter-t5vx9"
Sep 17 17:37:10.818146 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.817690    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/031c5c73-85cc-465f-b4c5-ac1b82042ac6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t5vx9\" (UID: \"031c5c73-85cc-465f-b4c5-ac1b82042ac6\") " pod="openshift-monitoring/node-exporter-t5vx9"
Sep 17 17:37:10.818146 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.817750    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/031c5c73-85cc-465f-b4c5-ac1b82042ac6-node-exporter-wtmp\") pod \"node-exporter-t5vx9\" (UID: \"031c5c73-85cc-465f-b4c5-ac1b82042ac6\") " pod="openshift-monitoring/node-exporter-t5vx9"
Sep 17 17:37:10.818146 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.817774    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxdfx\" (UniqueName: \"kubernetes.io/projected/031c5c73-85cc-465f-b4c5-ac1b82042ac6-kube-api-access-zxdfx\") pod \"node-exporter-t5vx9\" (UID: \"031c5c73-85cc-465f-b4c5-ac1b82042ac6\") " pod="openshift-monitoring/node-exporter-t5vx9"
Sep 17 17:37:10.818146 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.817799    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/031c5c73-85cc-465f-b4c5-ac1b82042ac6-node-exporter-textfile\") pod \"node-exporter-t5vx9\" (UID: \"031c5c73-85cc-465f-b4c5-ac1b82042ac6\") " pod="openshift-monitoring/node-exporter-t5vx9"
Sep 17 17:37:10.818146 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.817820    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/031c5c73-85cc-465f-b4c5-ac1b82042ac6-sys\") pod \"node-exporter-t5vx9\" (UID: \"031c5c73-85cc-465f-b4c5-ac1b82042ac6\") " pod="openshift-monitoring/node-exporter-t5vx9"
Sep 17 17:37:10.818146 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.817892    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/031c5c73-85cc-465f-b4c5-ac1b82042ac6-sys\") pod \"node-exporter-t5vx9\" (UID: \"031c5c73-85cc-465f-b4c5-ac1b82042ac6\") " pod="openshift-monitoring/node-exporter-t5vx9"
Sep 17 17:37:10.818146 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.817973    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/031c5c73-85cc-465f-b4c5-ac1b82042ac6-node-exporter-wtmp\") pod \"node-exporter-t5vx9\" (UID: \"031c5c73-85cc-465f-b4c5-ac1b82042ac6\") " pod="openshift-monitoring/node-exporter-t5vx9"
Sep 17 17:37:10.818146 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.818021    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/031c5c73-85cc-465f-b4c5-ac1b82042ac6-node-exporter-accelerators-collector-config\") pod \"node-exporter-t5vx9\" (UID: \"031c5c73-85cc-465f-b4c5-ac1b82042ac6\") " pod="openshift-monitoring/node-exporter-t5vx9"
Sep 17 17:37:10.818486 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.818174    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/031c5c73-85cc-465f-b4c5-ac1b82042ac6-metrics-client-ca\") pod \"node-exporter-t5vx9\" (UID: \"031c5c73-85cc-465f-b4c5-ac1b82042ac6\") " pod="openshift-monitoring/node-exporter-t5vx9"
Sep 17 17:37:10.819855 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.819835    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/031c5c73-85cc-465f-b4c5-ac1b82042ac6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t5vx9\" (UID: \"031c5c73-85cc-465f-b4c5-ac1b82042ac6\") " pod="openshift-monitoring/node-exporter-t5vx9"
Sep 17 17:37:10.819930 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.819912    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/031c5c73-85cc-465f-b4c5-ac1b82042ac6-node-exporter-tls\") pod \"node-exporter-t5vx9\" (UID: \"031c5c73-85cc-465f-b4c5-ac1b82042ac6\") " pod="openshift-monitoring/node-exporter-t5vx9"
Sep 17 17:37:10.829477 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.829456    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxdfx\" (UniqueName: \"kubernetes.io/projected/031c5c73-85cc-465f-b4c5-ac1b82042ac6-kube-api-access-zxdfx\") pod \"node-exporter-t5vx9\" (UID: \"031c5c73-85cc-465f-b4c5-ac1b82042ac6\") " pod="openshift-monitoring/node-exporter-t5vx9"
Sep 17 17:37:10.834432 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:10.834414    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t5vx9"
Sep 17 17:37:10.843479 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:37:10.843454    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031c5c73_85cc_465f_b4c5_ac1b82042ac6.slice/crio-ce4e5117300975f1c684924ea0d27f4cb0fbca52f63da1493e044672a203e591 WatchSource:0}: Error finding container ce4e5117300975f1c684924ea0d27f4cb0fbca52f63da1493e044672a203e591: Status 404 returned error can't find the container with id ce4e5117300975f1c684924ea0d27f4cb0fbca52f63da1493e044672a203e591
Sep 17 17:37:11.095262 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:11.095239    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-qn64b"]
Sep 17 17:37:11.098222 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:37:11.098199    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod215cb362_1c54_4086_83a8_99c96e59e90d.slice/crio-0322aae78ed320aacd500499f50c7a6b062929b748247f247e1defc1bf66435f WatchSource:0}: Error finding container 0322aae78ed320aacd500499f50c7a6b062929b748247f247e1defc1bf66435f: Status 404 returned error can't find the container with id 0322aae78ed320aacd500499f50c7a6b062929b748247f247e1defc1bf66435f
Sep 17 17:37:11.543119 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:11.543041    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t5vx9" event={"ID":"031c5c73-85cc-465f-b4c5-ac1b82042ac6","Type":"ContainerStarted","Data":"ce4e5117300975f1c684924ea0d27f4cb0fbca52f63da1493e044672a203e591"}
Sep 17 17:37:11.544464 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:11.544439    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-qn64b" event={"ID":"215cb362-1c54-4086-83a8-99c96e59e90d","Type":"ContainerStarted","Data":"0322aae78ed320aacd500499f50c7a6b062929b748247f247e1defc1bf66435f"}
Sep 17 17:37:12.139513 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:12.139429    2583 patch_prober.go:28] interesting pod/image-registry-7bd7db6998-stqrm container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 17:37:12.139685 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:12.139502    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7bd7db6998-stqrm" podUID="ff45a922-492d-4d9b-9039-c3a7759481a9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:37:12.550900 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:12.550824    2583 generic.go:356] "Generic (PLEG): container finished" podID="031c5c73-85cc-465f-b4c5-ac1b82042ac6" containerID="a1eae14c04334c83e8516436c149827157538b8192012b974dc5a7931a03d4fc" exitCode=0
Sep 17 17:37:12.551477 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:12.551357    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t5vx9" event={"ID":"031c5c73-85cc-465f-b4c5-ac1b82042ac6","Type":"ContainerDied","Data":"a1eae14c04334c83e8516436c149827157538b8192012b974dc5a7931a03d4fc"}
Sep 17 17:37:12.556215 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:12.555886    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-qn64b" event={"ID":"215cb362-1c54-4086-83a8-99c96e59e90d","Type":"ContainerStarted","Data":"a2c0352cbb7b8996524ffbb17d8204158636846211b4b550d621e541cb13c6bf"}
Sep 17 17:37:12.556215 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:12.555949    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-qn64b" event={"ID":"215cb362-1c54-4086-83a8-99c96e59e90d","Type":"ContainerStarted","Data":"f15efbe324a7f7cb4a8087f655f28698b924d7341ce160d80c473cc235bbae61"}
Sep 17 17:37:13.561591 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:13.561562    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t5vx9" event={"ID":"031c5c73-85cc-465f-b4c5-ac1b82042ac6","Type":"ContainerStarted","Data":"a354c7c113dc4aeda000a1a06b5d7c09c6d56d527b6cdba97977f4dedf39dca1"}
Sep 17 17:37:13.562057 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:13.561610    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t5vx9" event={"ID":"031c5c73-85cc-465f-b4c5-ac1b82042ac6","Type":"ContainerStarted","Data":"ddf873e3bcf8ef92b09a07e701344af090b594df629fd5f5b9685cdc3cb4a62b"}
Sep 17 17:37:13.563628 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:13.563602    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-qn64b" event={"ID":"215cb362-1c54-4086-83a8-99c96e59e90d","Type":"ContainerStarted","Data":"81f504dc1aecce937c8a9396aaee695fd7b7931fc33810ce1c1bab4f6c9d8c6d"}
Sep 17 17:37:13.590023 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:13.589978    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-t5vx9" podStartSLOduration=2.656969887 podStartE2EDuration="3.589963745s" totalImagesPullingTime="858.007432ms" totalInitContainerRuntime="74.986426ms" isStatefulPod=true podCreationTimestamp="2026-09-17 17:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:13.589198412 +0000 UTC m=+144.680505196" watchObservedRunningTime="2026-09-17 17:37:13.589963745 +0000 UTC m=+144.681270533"
Sep 17 17:37:13.612718 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:13.612678    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-qn64b" podStartSLOduration=2.282522125 podStartE2EDuration="3.61264716s" totalImagesPullingTime="1.330125035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:13.611360514 +0000 UTC m=+144.702667288" watchObservedRunningTime="2026-09-17 17:37:13.61264716 +0000 UTC m=+144.703953924"
Sep 17 17:37:22.138114 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:22.138079    2583 patch_prober.go:28] interesting pod/image-registry-7bd7db6998-stqrm container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 17:37:22.138492 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:22.138151    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7bd7db6998-stqrm" podUID="ff45a922-492d-4d9b-9039-c3a7759481a9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:37:27.166950 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.166913    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7bd7db6998-stqrm" podUID="ff45a922-492d-4d9b-9039-c3a7759481a9" containerName="registry" containerID="cri-o://c095a4f9a92da0134681f03ffe34256be370a74239305cc820501f8afb6b5432" gracePeriod=30
Sep 17 17:37:27.403438 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.403411    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:37:27.431045 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.430961    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff45a922-492d-4d9b-9039-c3a7759481a9-ca-trust-extracted\") pod \"ff45a922-492d-4d9b-9039-c3a7759481a9\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") "
Sep 17 17:37:27.431045 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.431007    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-bound-sa-token\") pod \"ff45a922-492d-4d9b-9039-c3a7759481a9\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") "
Sep 17 17:37:27.431251 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.431049    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff45a922-492d-4d9b-9039-c3a7759481a9-registry-certificates\") pod \"ff45a922-492d-4d9b-9039-c3a7759481a9\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") "
Sep 17 17:37:27.431251 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.431105    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tk97h\" (UniqueName: \"kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-kube-api-access-tk97h\") pod \"ff45a922-492d-4d9b-9039-c3a7759481a9\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") "
Sep 17 17:37:27.431251 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.431143    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff45a922-492d-4d9b-9039-c3a7759481a9-image-registry-private-configuration\") pod \"ff45a922-492d-4d9b-9039-c3a7759481a9\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") "
Sep 17 17:37:27.431251 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.431170    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff45a922-492d-4d9b-9039-c3a7759481a9-installation-pull-secrets\") pod \"ff45a922-492d-4d9b-9039-c3a7759481a9\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") "
Sep 17 17:37:27.431251 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.431200    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-registry-tls\") pod \"ff45a922-492d-4d9b-9039-c3a7759481a9\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") "
Sep 17 17:37:27.431251 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.431225    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff45a922-492d-4d9b-9039-c3a7759481a9-trusted-ca\") pod \"ff45a922-492d-4d9b-9039-c3a7759481a9\" (UID: \"ff45a922-492d-4d9b-9039-c3a7759481a9\") "
Sep 17 17:37:27.431773 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.431566    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff45a922-492d-4d9b-9039-c3a7759481a9-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ff45a922-492d-4d9b-9039-c3a7759481a9" (UID: "ff45a922-492d-4d9b-9039-c3a7759481a9"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:37:27.432175 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.431808    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff45a922-492d-4d9b-9039-c3a7759481a9-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ff45a922-492d-4d9b-9039-c3a7759481a9" (UID: "ff45a922-492d-4d9b-9039-c3a7759481a9"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:37:27.434507 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.434483    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-kube-api-access-tk97h" (OuterVolumeSpecName: "kube-api-access-tk97h") pod "ff45a922-492d-4d9b-9039-c3a7759481a9" (UID: "ff45a922-492d-4d9b-9039-c3a7759481a9"). InnerVolumeSpecName "kube-api-access-tk97h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:37:27.434609 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.434493    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff45a922-492d-4d9b-9039-c3a7759481a9-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ff45a922-492d-4d9b-9039-c3a7759481a9" (UID: "ff45a922-492d-4d9b-9039-c3a7759481a9"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:37:27.434734 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.434549    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff45a922-492d-4d9b-9039-c3a7759481a9-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ff45a922-492d-4d9b-9039-c3a7759481a9" (UID: "ff45a922-492d-4d9b-9039-c3a7759481a9"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:37:27.434734 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.434555    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ff45a922-492d-4d9b-9039-c3a7759481a9" (UID: "ff45a922-492d-4d9b-9039-c3a7759481a9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:37:27.435002 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.434973    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ff45a922-492d-4d9b-9039-c3a7759481a9" (UID: "ff45a922-492d-4d9b-9039-c3a7759481a9"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:37:27.443293 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.443265    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff45a922-492d-4d9b-9039-c3a7759481a9-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ff45a922-492d-4d9b-9039-c3a7759481a9" (UID: "ff45a922-492d-4d9b-9039-c3a7759481a9"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:37:27.532760 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.532734    2583 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff45a922-492d-4d9b-9039-c3a7759481a9-ca-trust-extracted\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 17:37:27.532760 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.532758    2583 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-bound-sa-token\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 17:37:27.532760 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.532768    2583 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff45a922-492d-4d9b-9039-c3a7759481a9-registry-certificates\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 17:37:27.532971 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.532776    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk97h\" (UniqueName: \"kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-kube-api-access-tk97h\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 17:37:27.532971 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.532785    2583 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff45a922-492d-4d9b-9039-c3a7759481a9-image-registry-private-configuration\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 17:37:27.532971 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.532794    2583 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff45a922-492d-4d9b-9039-c3a7759481a9-installation-pull-secrets\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 17:37:27.532971 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.532802    2583 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff45a922-492d-4d9b-9039-c3a7759481a9-registry-tls\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 17:37:27.532971 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.532810    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff45a922-492d-4d9b-9039-c3a7759481a9-trusted-ca\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 17:37:27.612770 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.612737    2583 generic.go:356] "Generic (PLEG): container finished" podID="ff45a922-492d-4d9b-9039-c3a7759481a9" containerID="c095a4f9a92da0134681f03ffe34256be370a74239305cc820501f8afb6b5432" exitCode=0
Sep 17 17:37:27.612904 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.612793    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bd7db6998-stqrm" event={"ID":"ff45a922-492d-4d9b-9039-c3a7759481a9","Type":"ContainerDied","Data":"c095a4f9a92da0134681f03ffe34256be370a74239305cc820501f8afb6b5432"}
Sep 17 17:37:27.612904 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.612797    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bd7db6998-stqrm"
Sep 17 17:37:27.612904 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.612859    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bd7db6998-stqrm" event={"ID":"ff45a922-492d-4d9b-9039-c3a7759481a9","Type":"ContainerDied","Data":"500d3a738ed9a5bfaf5159291ef713dad8622d4f47aca63e45bd98058a8de912"}
Sep 17 17:37:27.612904 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.612879    2583 scope.go:117] "RemoveContainer" containerID="c095a4f9a92da0134681f03ffe34256be370a74239305cc820501f8afb6b5432"
Sep 17 17:37:27.621068 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.621052    2583 scope.go:117] "RemoveContainer" containerID="c095a4f9a92da0134681f03ffe34256be370a74239305cc820501f8afb6b5432"
Sep 17 17:37:27.621405 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:37:27.621382    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c095a4f9a92da0134681f03ffe34256be370a74239305cc820501f8afb6b5432\": container with ID starting with c095a4f9a92da0134681f03ffe34256be370a74239305cc820501f8afb6b5432 not found: ID does not exist" containerID="c095a4f9a92da0134681f03ffe34256be370a74239305cc820501f8afb6b5432"
Sep 17 17:37:27.621457 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.621413    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c095a4f9a92da0134681f03ffe34256be370a74239305cc820501f8afb6b5432"} err="failed to get container status \"c095a4f9a92da0134681f03ffe34256be370a74239305cc820501f8afb6b5432\": rpc error: code = NotFound desc = could not find container \"c095a4f9a92da0134681f03ffe34256be370a74239305cc820501f8afb6b5432\": container with ID starting with c095a4f9a92da0134681f03ffe34256be370a74239305cc820501f8afb6b5432 not found: ID does not exist"
Sep 17 17:37:27.634186 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.634164    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7bd7db6998-stqrm"]
Sep 17 17:37:27.639014 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:27.638993    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7bd7db6998-stqrm"]
Sep 17 17:37:28.360053 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:37:28.360010    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-d6z4r" podUID="dad536d7-d27e-4d63-96e9-a6654126532d"
Sep 17 17:37:28.617122 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:28.617047    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d6z4r"
Sep 17 17:37:29.039268 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:29.039193    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff45a922-492d-4d9b-9039-c3a7759481a9" path="/var/lib/kubelet/pods/ff45a922-492d-4d9b-9039-c3a7759481a9/volumes"
Sep 17 17:37:33.271229 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:33.271149    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dad536d7-d27e-4d63-96e9-a6654126532d-metrics-tls\") pod \"dns-default-d6z4r\" (UID: \"dad536d7-d27e-4d63-96e9-a6654126532d\") " pod="openshift-dns/dns-default-d6z4r"
Sep 17 17:37:33.273403 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:33.273383    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dad536d7-d27e-4d63-96e9-a6654126532d-metrics-tls\") pod \"dns-default-d6z4r\" (UID: \"dad536d7-d27e-4d63-96e9-a6654126532d\") " pod="openshift-dns/dns-default-d6z4r"
Sep 17 17:37:33.420390 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:33.420364    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dng7x\""
Sep 17 17:37:33.428342 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:33.428324    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d6z4r"
Sep 17 17:37:33.701747 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:33.701558    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d6z4r"]
Sep 17 17:37:33.704136 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:37:33.704110    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad536d7_d27e_4d63_96e9_a6654126532d.slice/crio-1c3f1aa72073546c2fd1b6ba942c72ae000a7f161359098a4f5b90c9bc544f71 WatchSource:0}: Error finding container 1c3f1aa72073546c2fd1b6ba942c72ae000a7f161359098a4f5b90c9bc544f71: Status 404 returned error can't find the container with id 1c3f1aa72073546c2fd1b6ba942c72ae000a7f161359098a4f5b90c9bc544f71
Sep 17 17:37:34.639056 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:34.639018    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d6z4r" event={"ID":"dad536d7-d27e-4d63-96e9-a6654126532d","Type":"ContainerStarted","Data":"1c3f1aa72073546c2fd1b6ba942c72ae000a7f161359098a4f5b90c9bc544f71"}
Sep 17 17:37:35.644057 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:35.644028    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d6z4r" event={"ID":"dad536d7-d27e-4d63-96e9-a6654126532d","Type":"ContainerStarted","Data":"307cb5b1830d3e6e8ee49a98a4ebbe2aa465fd2545487ce2ff644e954b807772"}
Sep 17 17:37:35.644474 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:35.644078    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d6z4r" event={"ID":"dad536d7-d27e-4d63-96e9-a6654126532d","Type":"ContainerStarted","Data":"5ba3dc525defa0237ba7e94782ff110b5aeb41441290880b7684b0e609696956"}
Sep 17 17:37:35.644474 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:35.644174    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-d6z4r"
Sep 17 17:37:35.667840 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:35.667798    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-d6z4r" podStartSLOduration=129.047717396 podStartE2EDuration="2m10.667782741s" totalImagesPullingTime="1.620065345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:35.666548046 +0000 UTC m=+166.757854830" watchObservedRunningTime="2026-09-17 17:37:35.667782741 +0000 UTC m=+166.759089506"
Sep 17 17:37:40.145177 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.145140    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55fdb5bc84-wcdv9"]
Sep 17 17:37:40.145550 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.145401    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff45a922-492d-4d9b-9039-c3a7759481a9" containerName="registry"
Sep 17 17:37:40.145550 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.145413    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff45a922-492d-4d9b-9039-c3a7759481a9" containerName="registry"
Sep 17 17:37:40.145550 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.145459    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff45a922-492d-4d9b-9039-c3a7759481a9" containerName="registry"
Sep 17 17:37:40.148151 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.148133    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55fdb5bc84-wcdv9"
Sep 17 17:37:40.150456 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.150433    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-zn79t\""
Sep 17 17:37:40.152220 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.152194    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 17:37:40.152322 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.152292    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 17:37:40.152393 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.152321    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 17:37:40.152393 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.152294    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 17:37:40.152685 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.152654    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 17:37:40.153179 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.153162    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 17:37:40.154136 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.154119    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 17:37:40.157690 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.157673    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 17:37:40.163501 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.163480    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55fdb5bc84-wcdv9"]
Sep 17 17:37:40.218894 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.218870    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cb2c525-9d62-4026-b2b0-7d6760851b7c-trusted-ca-bundle\") pod \"console-55fdb5bc84-wcdv9\" (UID: \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\") " pod="openshift-console/console-55fdb5bc84-wcdv9"
Sep 17 17:37:40.219062 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.218907    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4cb2c525-9d62-4026-b2b0-7d6760851b7c-console-serving-cert\") pod \"console-55fdb5bc84-wcdv9\" (UID: \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\") " pod="openshift-console/console-55fdb5bc84-wcdv9"
Sep 17 17:37:40.219062 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.218924    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4cb2c525-9d62-4026-b2b0-7d6760851b7c-console-config\") pod \"console-55fdb5bc84-wcdv9\" (UID: \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\") " pod="openshift-console/console-55fdb5bc84-wcdv9"
Sep 17 17:37:40.219062 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.219005    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-647tr\" (UniqueName: \"kubernetes.io/projected/4cb2c525-9d62-4026-b2b0-7d6760851b7c-kube-api-access-647tr\") pod \"console-55fdb5bc84-wcdv9\" (UID: \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\") " pod="openshift-console/console-55fdb5bc84-wcdv9"
Sep 17 17:37:40.219062 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.219044    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4cb2c525-9d62-4026-b2b0-7d6760851b7c-console-oauth-config\") pod \"console-55fdb5bc84-wcdv9\" (UID: \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\") " pod="openshift-console/console-55fdb5bc84-wcdv9"
Sep 17 17:37:40.219238 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.219095    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4cb2c525-9d62-4026-b2b0-7d6760851b7c-service-ca\") pod \"console-55fdb5bc84-wcdv9\" (UID: \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\") " pod="openshift-console/console-55fdb5bc84-wcdv9"
Sep 17 17:37:40.219238 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.219125    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4cb2c525-9d62-4026-b2b0-7d6760851b7c-oauth-serving-cert\") pod \"console-55fdb5bc84-wcdv9\" (UID: \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\") " pod="openshift-console/console-55fdb5bc84-wcdv9"
Sep 17 17:37:40.320313 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.320274    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cb2c525-9d62-4026-b2b0-7d6760851b7c-trusted-ca-bundle\") pod \"console-55fdb5bc84-wcdv9\" (UID: \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\") " pod="openshift-console/console-55fdb5bc84-wcdv9"
Sep 17 17:37:40.320492 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.320331    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4cb2c525-9d62-4026-b2b0-7d6760851b7c-console-serving-cert\") pod \"console-55fdb5bc84-wcdv9\" (UID: \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\") " pod="openshift-console/console-55fdb5bc84-wcdv9"
Sep 17 17:37:40.320492 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.320355    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4cb2c525-9d62-4026-b2b0-7d6760851b7c-console-config\") pod \"console-55fdb5bc84-wcdv9\" (UID: \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\") " pod="openshift-console/console-55fdb5bc84-wcdv9"
Sep 17 17:37:40.320492 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.320385    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-647tr\" (UniqueName: \"kubernetes.io/projected/4cb2c525-9d62-4026-b2b0-7d6760851b7c-kube-api-access-647tr\") pod \"console-55fdb5bc84-wcdv9\" (UID: \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\") " pod="openshift-console/console-55fdb5bc84-wcdv9"
Sep 17 17:37:40.320492 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.320413    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4cb2c525-9d62-4026-b2b0-7d6760851b7c-console-oauth-config\") pod \"console-55fdb5bc84-wcdv9\" (UID: \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\") " pod="openshift-console/console-55fdb5bc84-wcdv9"
Sep 17 17:37:40.320492 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.320436    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4cb2c525-9d62-4026-b2b0-7d6760851b7c-service-ca\") pod \"console-55fdb5bc84-wcdv9\" (UID: \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\") " pod="openshift-console/console-55fdb5bc84-wcdv9"
Sep 17 17:37:40.320492 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.320460    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4cb2c525-9d62-4026-b2b0-7d6760851b7c-oauth-serving-cert\") pod \"console-55fdb5bc84-wcdv9\" (UID: \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\") " pod="openshift-console/console-55fdb5bc84-wcdv9"
Sep 17 17:37:40.321240 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.321209    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4cb2c525-9d62-4026-b2b0-7d6760851b7c-console-config\") pod \"console-55fdb5bc84-wcdv9\" (UID: \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\") " pod="openshift-console/console-55fdb5bc84-wcdv9"
Sep 17 17:37:40.321240 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.321229    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cb2c525-9d62-4026-b2b0-7d6760851b7c-trusted-ca-bundle\") pod \"console-55fdb5bc84-wcdv9\" (UID: \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\") " pod="openshift-console/console-55fdb5bc84-wcdv9"
Sep 17 17:37:40.321554 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.321539    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4cb2c525-9d62-4026-b2b0-7d6760851b7c-service-ca\") pod \"console-55fdb5bc84-wcdv9\" (UID: \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\") " pod="openshift-console/console-55fdb5bc84-wcdv9"
Sep 17 17:37:40.321791 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.321770    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4cb2c525-9d62-4026-b2b0-7d6760851b7c-oauth-serving-cert\") pod \"console-55fdb5bc84-wcdv9\" (UID: \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\") " pod="openshift-console/console-55fdb5bc84-wcdv9"
Sep 17 17:37:40.322698 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.322675    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4cb2c525-9d62-4026-b2b0-7d6760851b7c-console-oauth-config\") pod \"console-55fdb5bc84-wcdv9\" (UID: \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\") " pod="openshift-console/console-55fdb5bc84-wcdv9"
Sep 17 17:37:40.322886 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.322869    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4cb2c525-9d62-4026-b2b0-7d6760851b7c-console-serving-cert\") pod \"console-55fdb5bc84-wcdv9\" (UID: \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\") " pod="openshift-console/console-55fdb5bc84-wcdv9"
Sep 17 17:37:40.328956 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.328934    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-647tr\" (UniqueName: \"kubernetes.io/projected/4cb2c525-9d62-4026-b2b0-7d6760851b7c-kube-api-access-647tr\") pod \"console-55fdb5bc84-wcdv9\" (UID: \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\") " pod="openshift-console/console-55fdb5bc84-wcdv9"
Sep 17 17:37:40.457476 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.457393    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55fdb5bc84-wcdv9"
Sep 17 17:37:40.731468 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:40.731400    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55fdb5bc84-wcdv9"]
Sep 17 17:37:40.734484 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:37:40.734458    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cb2c525_9d62_4026_b2b0_7d6760851b7c.slice/crio-c83708f55e88ef69732580d4ec68b6d10f5665ea8d126db45a000617afdbf156 WatchSource:0}: Error finding container c83708f55e88ef69732580d4ec68b6d10f5665ea8d126db45a000617afdbf156: Status 404 returned error can't find the container with id c83708f55e88ef69732580d4ec68b6d10f5665ea8d126db45a000617afdbf156
Sep 17 17:37:41.668111 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:41.668001    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55fdb5bc84-wcdv9" event={"ID":"4cb2c525-9d62-4026-b2b0-7d6760851b7c","Type":"ContainerStarted","Data":"c83708f55e88ef69732580d4ec68b6d10f5665ea8d126db45a000617afdbf156"}
Sep 17 17:37:44.680374 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:44.680331    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55fdb5bc84-wcdv9" event={"ID":"4cb2c525-9d62-4026-b2b0-7d6760851b7c","Type":"ContainerStarted","Data":"e03f3bfb48a9dd8edab059bf80d5b8f4e624dda36fa142c700489de74d51ab35"}
Sep 17 17:37:44.704162 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:44.704114    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55fdb5bc84-wcdv9" podStartSLOduration=0.902290184 podStartE2EDuration="4.704097237s" totalImagesPullingTime="3.801807053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:37:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:44.703357425 +0000 UTC m=+175.794664219" watchObservedRunningTime="2026-09-17 17:37:44.704097237 +0000 UTC m=+175.795404004"
Sep 17 17:37:45.650793 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:45.650767    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-d6z4r"
Sep 17 17:37:50.457486 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:50.457449    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55fdb5bc84-wcdv9"
Sep 17 17:37:50.457972 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:50.457551    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55fdb5bc84-wcdv9"
Sep 17 17:37:50.462713 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:50.462689    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55fdb5bc84-wcdv9"
Sep 17 17:37:50.704428 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:50.704401    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55fdb5bc84-wcdv9"
Sep 17 17:37:55.721353 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:55.721318    2583 generic.go:356] "Generic (PLEG): container finished" podID="38abcd71-ad24-4ef8-b192-6124f30e33fd" containerID="493e5fa99c4c4448f0dcc9de5b139333d47db943668ba55ceba39815c033dcba" exitCode=0
Sep 17 17:37:55.721880 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:55.721698    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ntb24" event={"ID":"38abcd71-ad24-4ef8-b192-6124f30e33fd","Type":"ContainerDied","Data":"493e5fa99c4c4448f0dcc9de5b139333d47db943668ba55ceba39815c033dcba"}
Sep 17 17:37:55.722886 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:55.722855    2583 scope.go:117] "RemoveContainer" containerID="493e5fa99c4c4448f0dcc9de5b139333d47db943668ba55ceba39815c033dcba"
Sep 17 17:37:56.727169 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:56.727137    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-ntb24" event={"ID":"38abcd71-ad24-4ef8-b192-6124f30e33fd","Type":"ContainerStarted","Data":"a9bc5260ed2757954a3fca4f153ec3756d91716fb17533b4e4440a34ee65666a"}
Sep 17 17:37:57.731805 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:57.731774    2583 generic.go:356] "Generic (PLEG): container finished" podID="70fb0a3b-a013-4441-9403-d2905c25d596" containerID="7682bd9c1239bed3cd2c11020d5a16fed95b318eddf78861206a6c422420d964" exitCode=0
Sep 17 17:37:57.732195 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:57.731850    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-28g6n" event={"ID":"70fb0a3b-a013-4441-9403-d2905c25d596","Type":"ContainerDied","Data":"7682bd9c1239bed3cd2c11020d5a16fed95b318eddf78861206a6c422420d964"}
Sep 17 17:37:57.732195 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:57.732171    2583 scope.go:117] "RemoveContainer" containerID="7682bd9c1239bed3cd2c11020d5a16fed95b318eddf78861206a6c422420d964"
Sep 17 17:37:58.737245 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:37:58.737209    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-28g6n" event={"ID":"70fb0a3b-a013-4441-9403-d2905c25d596","Type":"ContainerStarted","Data":"cbfcd861022ea5794295fde43d9592f570ad9de3d75b48b1e5c44bd87f7ba07f"}
Sep 17 17:38:42.141054 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:38:42.141022    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55fdb5bc84-wcdv9"]
Sep 17 17:39:07.160133 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:07.160052    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55fdb5bc84-wcdv9" podUID="4cb2c525-9d62-4026-b2b0-7d6760851b7c" containerName="console" containerID="cri-o://e03f3bfb48a9dd8edab059bf80d5b8f4e624dda36fa142c700489de74d51ab35" gracePeriod=15
Sep 17 17:39:07.395209 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:07.395184    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55fdb5bc84-wcdv9"
Sep 17 17:39:07.448186 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:07.448125    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4cb2c525-9d62-4026-b2b0-7d6760851b7c-console-serving-cert\") pod \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\" (UID: \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\") "
Sep 17 17:39:07.448320 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:07.448196    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-647tr\" (UniqueName: \"kubernetes.io/projected/4cb2c525-9d62-4026-b2b0-7d6760851b7c-kube-api-access-647tr\") pod \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\" (UID: \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\") "
Sep 17 17:39:07.448384 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:07.448361    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4cb2c525-9d62-4026-b2b0-7d6760851b7c-oauth-serving-cert\") pod \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\" (UID: \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\") "
Sep 17 17:39:07.448441 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:07.448421    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4cb2c525-9d62-4026-b2b0-7d6760851b7c-service-ca\") pod \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\" (UID: \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\") "
Sep 17 17:39:07.448494 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:07.448453    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4cb2c525-9d62-4026-b2b0-7d6760851b7c-console-oauth-config\") pod \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\" (UID: \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\") "
Sep 17 17:39:07.448494 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:07.448488    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4cb2c525-9d62-4026-b2b0-7d6760851b7c-console-config\") pod \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\" (UID: \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\") "
Sep 17 17:39:07.448579 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:07.448509    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cb2c525-9d62-4026-b2b0-7d6760851b7c-trusted-ca-bundle\") pod \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\" (UID: \"4cb2c525-9d62-4026-b2b0-7d6760851b7c\") "
Sep 17 17:39:07.448770 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:07.448723    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cb2c525-9d62-4026-b2b0-7d6760851b7c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4cb2c525-9d62-4026-b2b0-7d6760851b7c" (UID: "4cb2c525-9d62-4026-b2b0-7d6760851b7c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:39:07.448957 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:07.448934    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cb2c525-9d62-4026-b2b0-7d6760851b7c-service-ca" (OuterVolumeSpecName: "service-ca") pod "4cb2c525-9d62-4026-b2b0-7d6760851b7c" (UID: "4cb2c525-9d62-4026-b2b0-7d6760851b7c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:39:07.448957 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:07.448946    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cb2c525-9d62-4026-b2b0-7d6760851b7c-console-config" (OuterVolumeSpecName: "console-config") pod "4cb2c525-9d62-4026-b2b0-7d6760851b7c" (UID: "4cb2c525-9d62-4026-b2b0-7d6760851b7c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:39:07.449102 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:07.448997    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cb2c525-9d62-4026-b2b0-7d6760851b7c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4cb2c525-9d62-4026-b2b0-7d6760851b7c" (UID: "4cb2c525-9d62-4026-b2b0-7d6760851b7c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:39:07.450379 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:07.450356    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cb2c525-9d62-4026-b2b0-7d6760851b7c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4cb2c525-9d62-4026-b2b0-7d6760851b7c" (UID: "4cb2c525-9d62-4026-b2b0-7d6760851b7c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:39:07.450379 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:07.450365    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb2c525-9d62-4026-b2b0-7d6760851b7c-kube-api-access-647tr" (OuterVolumeSpecName: "kube-api-access-647tr") pod "4cb2c525-9d62-4026-b2b0-7d6760851b7c" (UID: "4cb2c525-9d62-4026-b2b0-7d6760851b7c"). InnerVolumeSpecName "kube-api-access-647tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:39:07.450568 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:07.450545    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cb2c525-9d62-4026-b2b0-7d6760851b7c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4cb2c525-9d62-4026-b2b0-7d6760851b7c" (UID: "4cb2c525-9d62-4026-b2b0-7d6760851b7c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:39:07.549642 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:07.549612    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-647tr\" (UniqueName: \"kubernetes.io/projected/4cb2c525-9d62-4026-b2b0-7d6760851b7c-kube-api-access-647tr\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 17:39:07.549642 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:07.549638    2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4cb2c525-9d62-4026-b2b0-7d6760851b7c-oauth-serving-cert\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 17:39:07.549830 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:07.549649    2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4cb2c525-9d62-4026-b2b0-7d6760851b7c-service-ca\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 17:39:07.549830 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:07.549674    2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4cb2c525-9d62-4026-b2b0-7d6760851b7c-console-oauth-config\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 17:39:07.549830 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:07.549683    2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4cb2c525-9d62-4026-b2b0-7d6760851b7c-console-config\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 17:39:07.549830 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:07.549691    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cb2c525-9d62-4026-b2b0-7d6760851b7c-trusted-ca-bundle\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 17:39:07.549830 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:07.549698    2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4cb2c525-9d62-4026-b2b0-7d6760851b7c-console-serving-cert\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 17:39:07.973513 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:07.973477    2583 generic.go:356] "Generic (PLEG): container finished" podID="4cb2c525-9d62-4026-b2b0-7d6760851b7c" containerID="e03f3bfb48a9dd8edab059bf80d5b8f4e624dda36fa142c700489de74d51ab35" exitCode=0
Sep 17 17:39:07.973706 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:07.973523    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55fdb5bc84-wcdv9"
Sep 17 17:39:07.973706 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:07.973567    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55fdb5bc84-wcdv9" event={"ID":"4cb2c525-9d62-4026-b2b0-7d6760851b7c","Type":"ContainerDied","Data":"e03f3bfb48a9dd8edab059bf80d5b8f4e624dda36fa142c700489de74d51ab35"}
Sep 17 17:39:07.973706 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:07.973612    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55fdb5bc84-wcdv9" event={"ID":"4cb2c525-9d62-4026-b2b0-7d6760851b7c","Type":"ContainerDied","Data":"c83708f55e88ef69732580d4ec68b6d10f5665ea8d126db45a000617afdbf156"}
Sep 17 17:39:07.973706 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:07.973632    2583 scope.go:117] "RemoveContainer" containerID="e03f3bfb48a9dd8edab059bf80d5b8f4e624dda36fa142c700489de74d51ab35"
Sep 17 17:39:07.981701 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:07.981680    2583 scope.go:117] "RemoveContainer" containerID="e03f3bfb48a9dd8edab059bf80d5b8f4e624dda36fa142c700489de74d51ab35"
Sep 17 17:39:07.981975 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:39:07.981958    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e03f3bfb48a9dd8edab059bf80d5b8f4e624dda36fa142c700489de74d51ab35\": container with ID starting with e03f3bfb48a9dd8edab059bf80d5b8f4e624dda36fa142c700489de74d51ab35 not found: ID does not exist" containerID="e03f3bfb48a9dd8edab059bf80d5b8f4e624dda36fa142c700489de74d51ab35"
Sep 17 17:39:07.982022 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:07.981980    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e03f3bfb48a9dd8edab059bf80d5b8f4e624dda36fa142c700489de74d51ab35"} err="failed to get container status \"e03f3bfb48a9dd8edab059bf80d5b8f4e624dda36fa142c700489de74d51ab35\": rpc error: code = NotFound desc = could not find container \"e03f3bfb48a9dd8edab059bf80d5b8f4e624dda36fa142c700489de74d51ab35\": container with ID starting with e03f3bfb48a9dd8edab059bf80d5b8f4e624dda36fa142c700489de74d51ab35 not found: ID does not exist"
Sep 17 17:39:07.993523 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:07.993502    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55fdb5bc84-wcdv9"]
Sep 17 17:39:07.997730 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:07.997711    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55fdb5bc84-wcdv9"]
Sep 17 17:39:09.038407 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:09.038379    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cb2c525-9d62-4026-b2b0-7d6760851b7c" path="/var/lib/kubelet/pods/4cb2c525-9d62-4026-b2b0-7d6760851b7c/volumes"
Sep 17 17:39:50.911243 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:39:50.911206    2583 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 17:43:22.034483 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.034452    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b5556f557-9nrrc"]
Sep 17 17:43:22.035062 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.034783    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cb2c525-9d62-4026-b2b0-7d6760851b7c" containerName="console"
Sep 17 17:43:22.035062 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.034798    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cb2c525-9d62-4026-b2b0-7d6760851b7c" containerName="console"
Sep 17 17:43:22.035062 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.034857    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cb2c525-9d62-4026-b2b0-7d6760851b7c" containerName="console"
Sep 17 17:43:22.037359 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.037340    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5556f557-9nrrc"
Sep 17 17:43:22.041270 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.041252    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 17:43:22.041690 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.041652    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 17:43:22.041997 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.041984    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 17:43:22.043001 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.042985    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 17:43:22.043545 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.043529    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 17:43:22.047124 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.047109    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 17:43:22.047289 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.047277    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 17:43:22.047963 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.047949    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-zn79t\""
Sep 17 17:43:22.085733 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.085710    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 17:43:22.144507 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.144485    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b5556f557-9nrrc"]
Sep 17 17:43:22.162587 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.162562    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aadab25-7d9c-4796-ba49-b1f3c0643282-trusted-ca-bundle\") pod \"console-b5556f557-9nrrc\" (UID: \"6aadab25-7d9c-4796-ba49-b1f3c0643282\") " pod="openshift-console/console-b5556f557-9nrrc"
Sep 17 17:43:22.162721 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.162605    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6aadab25-7d9c-4796-ba49-b1f3c0643282-console-oauth-config\") pod \"console-b5556f557-9nrrc\" (UID: \"6aadab25-7d9c-4796-ba49-b1f3c0643282\") " pod="openshift-console/console-b5556f557-9nrrc"
Sep 17 17:43:22.162721 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.162633    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6aadab25-7d9c-4796-ba49-b1f3c0643282-console-config\") pod \"console-b5556f557-9nrrc\" (UID: \"6aadab25-7d9c-4796-ba49-b1f3c0643282\") " pod="openshift-console/console-b5556f557-9nrrc"
Sep 17 17:43:22.162825 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.162710    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6aadab25-7d9c-4796-ba49-b1f3c0643282-service-ca\") pod \"console-b5556f557-9nrrc\" (UID: \"6aadab25-7d9c-4796-ba49-b1f3c0643282\") " pod="openshift-console/console-b5556f557-9nrrc"
Sep 17 17:43:22.162825 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.162770    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6aadab25-7d9c-4796-ba49-b1f3c0643282-console-serving-cert\") pod \"console-b5556f557-9nrrc\" (UID: \"6aadab25-7d9c-4796-ba49-b1f3c0643282\") " pod="openshift-console/console-b5556f557-9nrrc"
Sep 17 17:43:22.162825 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.162798    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgz9k\" (UniqueName: \"kubernetes.io/projected/6aadab25-7d9c-4796-ba49-b1f3c0643282-kube-api-access-rgz9k\") pod \"console-b5556f557-9nrrc\" (UID: \"6aadab25-7d9c-4796-ba49-b1f3c0643282\") " pod="openshift-console/console-b5556f557-9nrrc"
Sep 17 17:43:22.162937 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.162844    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6aadab25-7d9c-4796-ba49-b1f3c0643282-oauth-serving-cert\") pod \"console-b5556f557-9nrrc\" (UID: \"6aadab25-7d9c-4796-ba49-b1f3c0643282\") " pod="openshift-console/console-b5556f557-9nrrc"
Sep 17 17:43:22.263699 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.263648    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6aadab25-7d9c-4796-ba49-b1f3c0643282-console-oauth-config\") pod \"console-b5556f557-9nrrc\" (UID: \"6aadab25-7d9c-4796-ba49-b1f3c0643282\") " pod="openshift-console/console-b5556f557-9nrrc"
Sep 17 17:43:22.263699 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.263703    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6aadab25-7d9c-4796-ba49-b1f3c0643282-console-config\") pod \"console-b5556f557-9nrrc\" (UID: \"6aadab25-7d9c-4796-ba49-b1f3c0643282\") " pod="openshift-console/console-b5556f557-9nrrc"
Sep 17 17:43:22.263910 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.263719    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6aadab25-7d9c-4796-ba49-b1f3c0643282-service-ca\") pod \"console-b5556f557-9nrrc\" (UID: \"6aadab25-7d9c-4796-ba49-b1f3c0643282\") " pod="openshift-console/console-b5556f557-9nrrc"
Sep 17 17:43:22.263910 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.263748    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6aadab25-7d9c-4796-ba49-b1f3c0643282-console-serving-cert\") pod \"console-b5556f557-9nrrc\" (UID: \"6aadab25-7d9c-4796-ba49-b1f3c0643282\") " pod="openshift-console/console-b5556f557-9nrrc"
Sep 17 17:43:22.263910 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.263765    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgz9k\" (UniqueName: \"kubernetes.io/projected/6aadab25-7d9c-4796-ba49-b1f3c0643282-kube-api-access-rgz9k\") pod \"console-b5556f557-9nrrc\" (UID: \"6aadab25-7d9c-4796-ba49-b1f3c0643282\") " pod="openshift-console/console-b5556f557-9nrrc"
Sep 17 17:43:22.263910 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.263796    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6aadab25-7d9c-4796-ba49-b1f3c0643282-oauth-serving-cert\") pod \"console-b5556f557-9nrrc\" (UID: \"6aadab25-7d9c-4796-ba49-b1f3c0643282\") " pod="openshift-console/console-b5556f557-9nrrc"
Sep 17 17:43:22.263910 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.263843    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aadab25-7d9c-4796-ba49-b1f3c0643282-trusted-ca-bundle\") pod \"console-b5556f557-9nrrc\" (UID: \"6aadab25-7d9c-4796-ba49-b1f3c0643282\") " pod="openshift-console/console-b5556f557-9nrrc"
Sep 17 17:43:22.264601 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.264574    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6aadab25-7d9c-4796-ba49-b1f3c0643282-oauth-serving-cert\") pod \"console-b5556f557-9nrrc\" (UID: \"6aadab25-7d9c-4796-ba49-b1f3c0643282\") " pod="openshift-console/console-b5556f557-9nrrc"
Sep 17 17:43:22.264719 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.264605    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6aadab25-7d9c-4796-ba49-b1f3c0643282-console-config\") pod \"console-b5556f557-9nrrc\" (UID: \"6aadab25-7d9c-4796-ba49-b1f3c0643282\") " pod="openshift-console/console-b5556f557-9nrrc"
Sep 17 17:43:22.264719 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.264697    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6aadab25-7d9c-4796-ba49-b1f3c0643282-service-ca\") pod \"console-b5556f557-9nrrc\" (UID: \"6aadab25-7d9c-4796-ba49-b1f3c0643282\") " pod="openshift-console/console-b5556f557-9nrrc"
Sep 17 17:43:22.264859 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.264766    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aadab25-7d9c-4796-ba49-b1f3c0643282-trusted-ca-bundle\") pod \"console-b5556f557-9nrrc\" (UID: \"6aadab25-7d9c-4796-ba49-b1f3c0643282\") " pod="openshift-console/console-b5556f557-9nrrc"
Sep 17 17:43:22.265972 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.265951    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6aadab25-7d9c-4796-ba49-b1f3c0643282-console-oauth-config\") pod \"console-b5556f557-9nrrc\" (UID: \"6aadab25-7d9c-4796-ba49-b1f3c0643282\") " pod="openshift-console/console-b5556f557-9nrrc"
Sep 17 17:43:22.266080 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.266062    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6aadab25-7d9c-4796-ba49-b1f3c0643282-console-serving-cert\") pod \"console-b5556f557-9nrrc\" (UID: \"6aadab25-7d9c-4796-ba49-b1f3c0643282\") " pod="openshift-console/console-b5556f557-9nrrc"
Sep 17 17:43:22.275060 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.275039    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgz9k\" (UniqueName: \"kubernetes.io/projected/6aadab25-7d9c-4796-ba49-b1f3c0643282-kube-api-access-rgz9k\") pod \"console-b5556f557-9nrrc\" (UID: \"6aadab25-7d9c-4796-ba49-b1f3c0643282\") " pod="openshift-console/console-b5556f557-9nrrc"
Sep 17 17:43:22.346254 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.346231    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5556f557-9nrrc"
Sep 17 17:43:22.618036 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.617956    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b5556f557-9nrrc"]
Sep 17 17:43:22.620701 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:43:22.620674    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aadab25_7d9c_4796_ba49_b1f3c0643282.slice/crio-6d4006d63ec8c3acad703a144490766491c458c5d15d58680b4637cb6ba5c87c WatchSource:0}: Error finding container 6d4006d63ec8c3acad703a144490766491c458c5d15d58680b4637cb6ba5c87c: Status 404 returned error can't find the container with id 6d4006d63ec8c3acad703a144490766491c458c5d15d58680b4637cb6ba5c87c
Sep 17 17:43:22.848324 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.848288    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5556f557-9nrrc" event={"ID":"6aadab25-7d9c-4796-ba49-b1f3c0643282","Type":"ContainerStarted","Data":"e10286af93590f2014bde3ca0cb3b33ab8c950a783d09e637b2d7f6648c762ef"}
Sep 17 17:43:22.848503 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.848342    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5556f557-9nrrc" event={"ID":"6aadab25-7d9c-4796-ba49-b1f3c0643282","Type":"ContainerStarted","Data":"6d4006d63ec8c3acad703a144490766491c458c5d15d58680b4637cb6ba5c87c"}
Sep 17 17:43:22.869305 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:22.869190    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b5556f557-9nrrc" podStartSLOduration=0.869169776 podStartE2EDuration="869.169776ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:43:22.868497317 +0000 UTC m=+513.959804092" watchObservedRunningTime="2026-09-17 17:43:22.869169776 +0000 UTC m=+513.960476542"
Sep 17 17:43:32.346506 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:32.346472    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b5556f557-9nrrc"
Sep 17 17:43:32.346896 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:32.346523    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b5556f557-9nrrc"
Sep 17 17:43:32.351417 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:32.351398    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b5556f557-9nrrc"
Sep 17 17:43:32.885300 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:43:32.885269    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b5556f557-9nrrc"
Sep 17 17:49:22.153160 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:22.153126    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-f2xrl"]
Sep 17 17:49:22.156119 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:22.156102    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-f2xrl"
Sep 17 17:49:22.158578 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:22.158544    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 17:49:22.158578 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:22.158558    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-22x6x\""
Sep 17 17:49:22.158578 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:22.158545    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 17:49:22.158902 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:22.158887    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 17:49:22.172233 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:22.172208    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-f2xrl"]
Sep 17 17:49:22.280412 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:22.280384    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/0740e1ab-71d2-4b8b-bbaf-34a999e876e5-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-f2xrl\" (UID: \"0740e1ab-71d2-4b8b-bbaf-34a999e876e5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-f2xrl"
Sep 17 17:49:22.280412 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:22.280412    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7hkx\" (UniqueName: \"kubernetes.io/projected/0740e1ab-71d2-4b8b-bbaf-34a999e876e5-kube-api-access-m7hkx\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-f2xrl\" (UID: \"0740e1ab-71d2-4b8b-bbaf-34a999e876e5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-f2xrl"
Sep 17 17:49:22.381588 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:22.381562    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/0740e1ab-71d2-4b8b-bbaf-34a999e876e5-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-f2xrl\" (UID: \"0740e1ab-71d2-4b8b-bbaf-34a999e876e5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-f2xrl"
Sep 17 17:49:22.381588 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:22.381588    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7hkx\" (UniqueName: \"kubernetes.io/projected/0740e1ab-71d2-4b8b-bbaf-34a999e876e5-kube-api-access-m7hkx\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-f2xrl\" (UID: \"0740e1ab-71d2-4b8b-bbaf-34a999e876e5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-f2xrl"
Sep 17 17:49:22.383850 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:22.383828    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/0740e1ab-71d2-4b8b-bbaf-34a999e876e5-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-f2xrl\" (UID: \"0740e1ab-71d2-4b8b-bbaf-34a999e876e5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-f2xrl"
Sep 17 17:49:22.393675 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:22.393640    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7hkx\" (UniqueName: \"kubernetes.io/projected/0740e1ab-71d2-4b8b-bbaf-34a999e876e5-kube-api-access-m7hkx\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-f2xrl\" (UID: \"0740e1ab-71d2-4b8b-bbaf-34a999e876e5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-f2xrl"
Sep 17 17:49:22.466620 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:22.466560    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-f2xrl"
Sep 17 17:49:22.746349 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:22.746273    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-f2xrl"]
Sep 17 17:49:22.749290 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:49:22.749263    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0740e1ab_71d2_4b8b_bbaf_34a999e876e5.slice/crio-36d8f9a0a58befb333e818d393767002244bfa4e39a1b49e7146841042a7e117 WatchSource:0}: Error finding container 36d8f9a0a58befb333e818d393767002244bfa4e39a1b49e7146841042a7e117: Status 404 returned error can't find the container with id 36d8f9a0a58befb333e818d393767002244bfa4e39a1b49e7146841042a7e117
Sep 17 17:49:22.751249 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:22.751228    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:49:23.073256 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:23.073176    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-f2xrl" event={"ID":"0740e1ab-71d2-4b8b-bbaf-34a999e876e5","Type":"ContainerStarted","Data":"36d8f9a0a58befb333e818d393767002244bfa4e39a1b49e7146841042a7e117"}
Sep 17 17:49:26.087767 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:26.087730    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-f2xrl" event={"ID":"0740e1ab-71d2-4b8b-bbaf-34a999e876e5","Type":"ContainerStarted","Data":"9f49e1eda2eb60804cdb29f33ac6e86e3f7654a6825cd14c3aaf8790707f1d9b"}
Sep 17 17:49:26.088165 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:26.087867    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-f2xrl"
Sep 17 17:49:26.120171 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:26.120125    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-f2xrl" podStartSLOduration=0.913929669 podStartE2EDuration="4.120109736s" totalImagesPullingTime="3.206180067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:49:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:49:26.118756377 +0000 UTC m=+877.210063154" watchObservedRunningTime="2026-09-17 17:49:26.120109736 +0000 UTC m=+877.211416500"
Sep 17 17:49:27.722875 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:27.722846    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-2b2b9"]
Sep 17 17:49:27.725842 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:27.725824    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-2b2b9"
Sep 17 17:49:27.729041 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:27.729022    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 17 17:49:27.729329 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:27.729317    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 17:49:27.729942 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:27.729929    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-rk7tc\""
Sep 17 17:49:27.743908 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:27.743881    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-2b2b9"]
Sep 17 17:49:27.826054 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:27.826027    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xchqf\" (UniqueName: \"kubernetes.io/projected/8b57579f-008f-4449-820e-47ec4a05101c-kube-api-access-xchqf\") pod \"keda-admission-6c4d864b86-2b2b9\" (UID: \"8b57579f-008f-4449-820e-47ec4a05101c\") " pod="openshift-keda/keda-admission-6c4d864b86-2b2b9"
Sep 17 17:49:27.826195 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:27.826069    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8b57579f-008f-4449-820e-47ec4a05101c-certificates\") pod \"keda-admission-6c4d864b86-2b2b9\" (UID: \"8b57579f-008f-4449-820e-47ec4a05101c\") " pod="openshift-keda/keda-admission-6c4d864b86-2b2b9"
Sep 17 17:49:27.926570 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:27.926539    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xchqf\" (UniqueName: \"kubernetes.io/projected/8b57579f-008f-4449-820e-47ec4a05101c-kube-api-access-xchqf\") pod \"keda-admission-6c4d864b86-2b2b9\" (UID: \"8b57579f-008f-4449-820e-47ec4a05101c\") " pod="openshift-keda/keda-admission-6c4d864b86-2b2b9"
Sep 17 17:49:27.926722 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:27.926578    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8b57579f-008f-4449-820e-47ec4a05101c-certificates\") pod \"keda-admission-6c4d864b86-2b2b9\" (UID: \"8b57579f-008f-4449-820e-47ec4a05101c\") " pod="openshift-keda/keda-admission-6c4d864b86-2b2b9"
Sep 17 17:49:27.926765 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:49:27.926721    2583 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Sep 17 17:49:27.926765 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:49:27.926742    2583 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-2b2b9: secret "keda-admission-webhooks-certs" not found
Sep 17 17:49:27.926833 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:49:27.926790    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8b57579f-008f-4449-820e-47ec4a05101c-certificates podName:8b57579f-008f-4449-820e-47ec4a05101c nodeName:}" failed. No retries permitted until 2026-09-17 17:49:28.426774214 +0000 UTC m=+879.518080959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/8b57579f-008f-4449-820e-47ec4a05101c-certificates") pod "keda-admission-6c4d864b86-2b2b9" (UID: "8b57579f-008f-4449-820e-47ec4a05101c") : secret "keda-admission-webhooks-certs" not found
Sep 17 17:49:27.935900 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:27.935878    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xchqf\" (UniqueName: \"kubernetes.io/projected/8b57579f-008f-4449-820e-47ec4a05101c-kube-api-access-xchqf\") pod \"keda-admission-6c4d864b86-2b2b9\" (UID: \"8b57579f-008f-4449-820e-47ec4a05101c\") " pod="openshift-keda/keda-admission-6c4d864b86-2b2b9"
Sep 17 17:49:28.429372 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:28.429336    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8b57579f-008f-4449-820e-47ec4a05101c-certificates\") pod \"keda-admission-6c4d864b86-2b2b9\" (UID: \"8b57579f-008f-4449-820e-47ec4a05101c\") " pod="openshift-keda/keda-admission-6c4d864b86-2b2b9"
Sep 17 17:49:28.431968 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:28.431936    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8b57579f-008f-4449-820e-47ec4a05101c-certificates\") pod \"keda-admission-6c4d864b86-2b2b9\" (UID: \"8b57579f-008f-4449-820e-47ec4a05101c\") " pod="openshift-keda/keda-admission-6c4d864b86-2b2b9"
Sep 17 17:49:28.635905 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:28.635875    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-2b2b9"
Sep 17 17:49:28.907277 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:28.907236    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-2b2b9"]
Sep 17 17:49:28.909486 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:49:28.909456    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b57579f_008f_4449_820e_47ec4a05101c.slice/crio-676baa90713fd828c99f37994c3bfdaccf7896080498e8b309472d3afac2a0ce WatchSource:0}: Error finding container 676baa90713fd828c99f37994c3bfdaccf7896080498e8b309472d3afac2a0ce: Status 404 returned error can't find the container with id 676baa90713fd828c99f37994c3bfdaccf7896080498e8b309472d3afac2a0ce
Sep 17 17:49:29.100587 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:29.100557    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-2b2b9" event={"ID":"8b57579f-008f-4449-820e-47ec4a05101c","Type":"ContainerStarted","Data":"676baa90713fd828c99f37994c3bfdaccf7896080498e8b309472d3afac2a0ce"}
Sep 17 17:49:31.108386 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:31.108353    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-2b2b9" event={"ID":"8b57579f-008f-4449-820e-47ec4a05101c","Type":"ContainerStarted","Data":"56bd2f9848eae14fabd875702c45bdbc3c31a1f2fe7df400dd6e29d5731494fd"}
Sep 17 17:49:31.108778 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:31.108493    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-2b2b9"
Sep 17 17:49:31.129289 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:31.129245    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-2b2b9" podStartSLOduration=2.789114519 podStartE2EDuration="4.129230735s" totalImagesPullingTime="1.340116216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:49:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:49:31.128532762 +0000 UTC m=+882.219839546" watchObservedRunningTime="2026-09-17 17:49:31.129230735 +0000 UTC m=+882.220537597"
Sep 17 17:49:47.093510 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:47.093483    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-f2xrl"
Sep 17 17:49:52.114080 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:49:52.114049    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-2b2b9"
Sep 17 17:50:43.076860 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:50:43.076832    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6wvph"]
Sep 17 17:50:43.079763 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:50:43.079744    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6wvph"
Sep 17 17:50:43.082274 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:50:43.082257    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-jwkzc\""
Sep 17 17:50:43.083108 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:50:43.083092    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 17 17:50:43.083583 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:50:43.083563    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 17 17:50:43.090642 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:50:43.090621    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6wvph"]
Sep 17 17:50:43.152276 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:50:43.152248    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac86080e-4a30-4f5d-88ce-4a0732596cc2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6wvph\" (UID: \"ac86080e-4a30-4f5d-88ce-4a0732596cc2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6wvph"
Sep 17 17:50:43.152431 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:50:43.152328    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lplgx\" (UniqueName: \"kubernetes.io/projected/ac86080e-4a30-4f5d-88ce-4a0732596cc2-kube-api-access-lplgx\") pod \"cert-manager-webhook-6486ffd997-6wvph\" (UID: \"ac86080e-4a30-4f5d-88ce-4a0732596cc2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6wvph"
Sep 17 17:50:43.253163 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:50:43.253133    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lplgx\" (UniqueName: \"kubernetes.io/projected/ac86080e-4a30-4f5d-88ce-4a0732596cc2-kube-api-access-lplgx\") pod \"cert-manager-webhook-6486ffd997-6wvph\" (UID: \"ac86080e-4a30-4f5d-88ce-4a0732596cc2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6wvph"
Sep 17 17:50:43.253307 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:50:43.253218    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac86080e-4a30-4f5d-88ce-4a0732596cc2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6wvph\" (UID: \"ac86080e-4a30-4f5d-88ce-4a0732596cc2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6wvph"
Sep 17 17:50:43.261373 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:50:43.261353    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac86080e-4a30-4f5d-88ce-4a0732596cc2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6wvph\" (UID: \"ac86080e-4a30-4f5d-88ce-4a0732596cc2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6wvph"
Sep 17 17:50:43.261485 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:50:43.261399    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lplgx\" (UniqueName: \"kubernetes.io/projected/ac86080e-4a30-4f5d-88ce-4a0732596cc2-kube-api-access-lplgx\") pod \"cert-manager-webhook-6486ffd997-6wvph\" (UID: \"ac86080e-4a30-4f5d-88ce-4a0732596cc2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6wvph"
Sep 17 17:50:43.388759 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:50:43.388738    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6wvph"
Sep 17 17:50:43.659173 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:50:43.658979    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6wvph"]
Sep 17 17:50:43.661846 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:50:43.661817    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac86080e_4a30_4f5d_88ce_4a0732596cc2.slice/crio-39dfc897bb59a1be61de522139a0cd38d99b8ee25eb1d9c8d378c526518ad0f2 WatchSource:0}: Error finding container 39dfc897bb59a1be61de522139a0cd38d99b8ee25eb1d9c8d378c526518ad0f2: Status 404 returned error can't find the container with id 39dfc897bb59a1be61de522139a0cd38d99b8ee25eb1d9c8d378c526518ad0f2
Sep 17 17:50:44.376055 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:50:44.375996    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6wvph" event={"ID":"ac86080e-4a30-4f5d-88ce-4a0732596cc2","Type":"ContainerStarted","Data":"39dfc897bb59a1be61de522139a0cd38d99b8ee25eb1d9c8d378c526518ad0f2"}
Sep 17 17:50:47.388917 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:50:47.388888    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6wvph" event={"ID":"ac86080e-4a30-4f5d-88ce-4a0732596cc2","Type":"ContainerStarted","Data":"773d556418c19ae460d0c2bfe566811cf1248482e923cb006523bca4f8332c5b"}
Sep 17 17:50:47.389277 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:50:47.389021    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6wvph"
Sep 17 17:50:47.412349 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:50:47.408191    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-6wvph" podStartSLOduration=1.305752217 podStartE2EDuration="4.408168812s" totalImagesPullingTime="3.102416595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:50:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:50:47.408157778 +0000 UTC m=+958.499464546" watchObservedRunningTime="2026-09-17 17:50:47.408168812 +0000 UTC m=+958.499475578"
Sep 17 17:50:53.393877 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:50:53.393843    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6wvph"
Sep 17 17:51:06.367095 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:51:06.367011    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6796bd7874-c2qsq"]
Sep 17 17:51:06.370472 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:51:06.370448    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6796bd7874-c2qsq"
Sep 17 17:51:06.389502 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:51:06.389476    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:51:06.389617 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:51:06.389476    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 17 17:51:06.390103 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:51:06.390078    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 17:51:06.390219 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:51:06.390109    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 17 17:51:06.390219 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:51:06.390082    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tdsdt\""
Sep 17 17:51:06.390462 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:51:06.390079    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 17 17:51:06.393083 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:51:06.393058    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6796bd7874-c2qsq"]
Sep 17 17:51:06.525131 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:51:06.525102    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e4905595-4329-4cb7-928d-3145b47235ac-manager-config\") pod \"lws-controller-manager-6796bd7874-c2qsq\" (UID: \"e4905595-4329-4cb7-928d-3145b47235ac\") " pod="openshift-lws-operator/lws-controller-manager-6796bd7874-c2qsq"
Sep 17 17:51:06.525274 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:51:06.525146    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4905595-4329-4cb7-928d-3145b47235ac-metrics-cert\") pod \"lws-controller-manager-6796bd7874-c2qsq\" (UID: \"e4905595-4329-4cb7-928d-3145b47235ac\") " pod="openshift-lws-operator/lws-controller-manager-6796bd7874-c2qsq"
Sep 17 17:51:06.525274 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:51:06.525203    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk5ts\" (UniqueName: \"kubernetes.io/projected/e4905595-4329-4cb7-928d-3145b47235ac-kube-api-access-mk5ts\") pod \"lws-controller-manager-6796bd7874-c2qsq\" (UID: \"e4905595-4329-4cb7-928d-3145b47235ac\") " pod="openshift-lws-operator/lws-controller-manager-6796bd7874-c2qsq"
Sep 17 17:51:06.525274 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:51:06.525229    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4905595-4329-4cb7-928d-3145b47235ac-cert\") pod \"lws-controller-manager-6796bd7874-c2qsq\" (UID: \"e4905595-4329-4cb7-928d-3145b47235ac\") " pod="openshift-lws-operator/lws-controller-manager-6796bd7874-c2qsq"
Sep 17 17:51:06.626519 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:51:06.626441    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4905595-4329-4cb7-928d-3145b47235ac-metrics-cert\") pod \"lws-controller-manager-6796bd7874-c2qsq\" (UID: \"e4905595-4329-4cb7-928d-3145b47235ac\") " pod="openshift-lws-operator/lws-controller-manager-6796bd7874-c2qsq"
Sep 17 17:51:06.626519 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:51:06.626490    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk5ts\" (UniqueName: \"kubernetes.io/projected/e4905595-4329-4cb7-928d-3145b47235ac-kube-api-access-mk5ts\") pod \"lws-controller-manager-6796bd7874-c2qsq\" (UID: \"e4905595-4329-4cb7-928d-3145b47235ac\") " pod="openshift-lws-operator/lws-controller-manager-6796bd7874-c2qsq"
Sep 17 17:51:06.626763 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:51:06.626522    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4905595-4329-4cb7-928d-3145b47235ac-cert\") pod \"lws-controller-manager-6796bd7874-c2qsq\" (UID: \"e4905595-4329-4cb7-928d-3145b47235ac\") " pod="openshift-lws-operator/lws-controller-manager-6796bd7874-c2qsq"
Sep 17 17:51:06.626763 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:51:06.626569    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e4905595-4329-4cb7-928d-3145b47235ac-manager-config\") pod \"lws-controller-manager-6796bd7874-c2qsq\" (UID: \"e4905595-4329-4cb7-928d-3145b47235ac\") " pod="openshift-lws-operator/lws-controller-manager-6796bd7874-c2qsq"
Sep 17 17:51:06.627253 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:51:06.627230    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e4905595-4329-4cb7-928d-3145b47235ac-manager-config\") pod \"lws-controller-manager-6796bd7874-c2qsq\" (UID: \"e4905595-4329-4cb7-928d-3145b47235ac\") " pod="openshift-lws-operator/lws-controller-manager-6796bd7874-c2qsq"
Sep 17 17:51:06.629020 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:51:06.629001    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4905595-4329-4cb7-928d-3145b47235ac-metrics-cert\") pod \"lws-controller-manager-6796bd7874-c2qsq\" (UID: \"e4905595-4329-4cb7-928d-3145b47235ac\") " pod="openshift-lws-operator/lws-controller-manager-6796bd7874-c2qsq"
Sep 17 17:51:06.629100 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:51:06.629065    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4905595-4329-4cb7-928d-3145b47235ac-cert\") pod \"lws-controller-manager-6796bd7874-c2qsq\" (UID: \"e4905595-4329-4cb7-928d-3145b47235ac\") " pod="openshift-lws-operator/lws-controller-manager-6796bd7874-c2qsq"
Sep 17 17:51:06.652972 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:51:06.652947    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk5ts\" (UniqueName: \"kubernetes.io/projected/e4905595-4329-4cb7-928d-3145b47235ac-kube-api-access-mk5ts\") pod \"lws-controller-manager-6796bd7874-c2qsq\" (UID: \"e4905595-4329-4cb7-928d-3145b47235ac\") " pod="openshift-lws-operator/lws-controller-manager-6796bd7874-c2qsq"
Sep 17 17:51:06.678776 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:51:06.678752    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6796bd7874-c2qsq"
Sep 17 17:51:06.962618 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:51:06.962530    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6796bd7874-c2qsq"]
Sep 17 17:51:06.965348 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:51:06.965323    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4905595_4329_4cb7_928d_3145b47235ac.slice/crio-755c3573946424030466d47d8694999221c23cfa9c52f4062d7d15f95221ae47 WatchSource:0}: Error finding container 755c3573946424030466d47d8694999221c23cfa9c52f4062d7d15f95221ae47: Status 404 returned error can't find the container with id 755c3573946424030466d47d8694999221c23cfa9c52f4062d7d15f95221ae47
Sep 17 17:51:07.462610 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:51:07.462572    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6796bd7874-c2qsq" event={"ID":"e4905595-4329-4cb7-928d-3145b47235ac","Type":"ContainerStarted","Data":"755c3573946424030466d47d8694999221c23cfa9c52f4062d7d15f95221ae47"}
Sep 17 17:51:21.512958 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:51:21.512922    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6796bd7874-c2qsq" event={"ID":"e4905595-4329-4cb7-928d-3145b47235ac","Type":"ContainerStarted","Data":"b4b194f1c4009e721c1a3f70258dfe0586a1b31179473ee25ba1194f914163e5"}
Sep 17 17:51:21.513331 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:51:21.513019    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6796bd7874-c2qsq"
Sep 17 17:51:21.531805 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:51:21.531756    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6796bd7874-c2qsq" podStartSLOduration=1.733187826 podStartE2EDuration="15.531740987s" totalImagesPullingTime="13.798553161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:51:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:51:21.531357489 +0000 UTC m=+992.622664246" watchObservedRunningTime="2026-09-17 17:51:21.531740987 +0000 UTC m=+992.623047753"
Sep 17 17:51:32.518653 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:51:32.518575    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6796bd7874-c2qsq"
Sep 17 17:52:05.095767 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:05.095739    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-8j98m"]
Sep 17 17:52:05.103322 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:05.103301    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8j98m"
Sep 17 17:52:05.106681 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:05.106645    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 17:52:05.106756 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:05.106703    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-s4fpv\""
Sep 17 17:52:05.107290 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:05.107265    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 17:52:05.107395 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:05.107299    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 17 17:52:05.107395 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:05.107304    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 17 17:52:05.112152 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:05.112131    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-8j98m"]
Sep 17 17:52:05.240324 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:05.240292    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e5c0588a-302e-421f-be89-fe1b306900be-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-8j98m\" (UID: \"e5c0588a-302e-421f-be89-fe1b306900be\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8j98m"
Sep 17 17:52:05.240324 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:05.240327    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e5c0588a-302e-421f-be89-fe1b306900be-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-8j98m\" (UID: \"e5c0588a-302e-421f-be89-fe1b306900be\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8j98m"
Sep 17 17:52:05.240541 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:05.240418    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8nht\" (UniqueName: \"kubernetes.io/projected/e5c0588a-302e-421f-be89-fe1b306900be-kube-api-access-k8nht\") pod \"kuadrant-console-plugin-fbdd4f445-8j98m\" (UID: \"e5c0588a-302e-421f-be89-fe1b306900be\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8j98m"
Sep 17 17:52:05.341419 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:05.341391    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8nht\" (UniqueName: \"kubernetes.io/projected/e5c0588a-302e-421f-be89-fe1b306900be-kube-api-access-k8nht\") pod \"kuadrant-console-plugin-fbdd4f445-8j98m\" (UID: \"e5c0588a-302e-421f-be89-fe1b306900be\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8j98m"
Sep 17 17:52:05.341588 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:05.341441    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e5c0588a-302e-421f-be89-fe1b306900be-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-8j98m\" (UID: \"e5c0588a-302e-421f-be89-fe1b306900be\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8j98m"
Sep 17 17:52:05.341588 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:05.341459    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e5c0588a-302e-421f-be89-fe1b306900be-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-8j98m\" (UID: \"e5c0588a-302e-421f-be89-fe1b306900be\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8j98m"
Sep 17 17:52:05.342168 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:05.342145    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e5c0588a-302e-421f-be89-fe1b306900be-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-8j98m\" (UID: \"e5c0588a-302e-421f-be89-fe1b306900be\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8j98m"
Sep 17 17:52:05.343863 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:05.343842    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e5c0588a-302e-421f-be89-fe1b306900be-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-8j98m\" (UID: \"e5c0588a-302e-421f-be89-fe1b306900be\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8j98m"
Sep 17 17:52:05.359443 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:05.359386    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8nht\" (UniqueName: \"kubernetes.io/projected/e5c0588a-302e-421f-be89-fe1b306900be-kube-api-access-k8nht\") pod \"kuadrant-console-plugin-fbdd4f445-8j98m\" (UID: \"e5c0588a-302e-421f-be89-fe1b306900be\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8j98m"
Sep 17 17:52:05.413097 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:05.413078    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8j98m"
Sep 17 17:52:05.685616 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:05.685595    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-8j98m"]
Sep 17 17:52:05.688160 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:52:05.688132    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5c0588a_302e_421f_be89_fe1b306900be.slice/crio-3cfeba80895499075da098e81fa7ce5fe2d1a49b905c4ee59e8e94d60fb62a98 WatchSource:0}: Error finding container 3cfeba80895499075da098e81fa7ce5fe2d1a49b905c4ee59e8e94d60fb62a98: Status 404 returned error can't find the container with id 3cfeba80895499075da098e81fa7ce5fe2d1a49b905c4ee59e8e94d60fb62a98
Sep 17 17:52:06.689383 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:06.689322    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8j98m" event={"ID":"e5c0588a-302e-421f-be89-fe1b306900be","Type":"ContainerStarted","Data":"3cfeba80895499075da098e81fa7ce5fe2d1a49b905c4ee59e8e94d60fb62a98"}
Sep 17 17:52:10.710050 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:10.710016    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8j98m" event={"ID":"e5c0588a-302e-421f-be89-fe1b306900be","Type":"ContainerStarted","Data":"09f28ca4f4a49ebf22622199e2d492e6894a9b99fbd7c0a2e7319b2759eaf10e"}
Sep 17 17:52:10.736433 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:10.736388    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8j98m" podStartSLOduration=1.286682559 podStartE2EDuration="5.736373952s" totalImagesPullingTime="4.449691393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:52:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:52:10.735185925 +0000 UTC m=+1041.826492690" watchObservedRunningTime="2026-09-17 17:52:10.736373952 +0000 UTC m=+1041.827680717"
Sep 17 17:52:47.473553 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:47.473520    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6lknq"]
Sep 17 17:52:47.491184 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:47.491158    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-6lknq"
Sep 17 17:52:47.493941 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:47.493917    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 17 17:52:47.496815 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:47.496783    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6lknq"]
Sep 17 17:52:47.518701 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:47.518677    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6lknq"]
Sep 17 17:52:47.550322 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:47.550300    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9zmm\" (UniqueName: \"kubernetes.io/projected/0c053a59-a277-4666-8612-aae4851ddfae-kube-api-access-m9zmm\") pod \"limitador-limitador-84994f7ffc-6lknq\" (UID: \"0c053a59-a277-4666-8612-aae4851ddfae\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6lknq"
Sep 17 17:52:47.550422 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:47.550341    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0c053a59-a277-4666-8612-aae4851ddfae-config-file\") pod \"limitador-limitador-84994f7ffc-6lknq\" (UID: \"0c053a59-a277-4666-8612-aae4851ddfae\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6lknq"
Sep 17 17:52:47.651425 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:47.651390    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9zmm\" (UniqueName: \"kubernetes.io/projected/0c053a59-a277-4666-8612-aae4851ddfae-kube-api-access-m9zmm\") pod \"limitador-limitador-84994f7ffc-6lknq\" (UID: \"0c053a59-a277-4666-8612-aae4851ddfae\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6lknq"
Sep 17 17:52:47.651574 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:47.651436    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0c053a59-a277-4666-8612-aae4851ddfae-config-file\") pod \"limitador-limitador-84994f7ffc-6lknq\" (UID: \"0c053a59-a277-4666-8612-aae4851ddfae\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6lknq"
Sep 17 17:52:47.652038 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:47.652021    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0c053a59-a277-4666-8612-aae4851ddfae-config-file\") pod \"limitador-limitador-84994f7ffc-6lknq\" (UID: \"0c053a59-a277-4666-8612-aae4851ddfae\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6lknq"
Sep 17 17:52:47.659604 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:47.659578    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9zmm\" (UniqueName: \"kubernetes.io/projected/0c053a59-a277-4666-8612-aae4851ddfae-kube-api-access-m9zmm\") pod \"limitador-limitador-84994f7ffc-6lknq\" (UID: \"0c053a59-a277-4666-8612-aae4851ddfae\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6lknq"
Sep 17 17:52:47.801645 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:47.801573    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-6lknq"
Sep 17 17:52:47.878987 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:47.878963    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dc4ms"]
Sep 17 17:52:47.882414 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:47.882392    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-dc4ms"
Sep 17 17:52:47.884797 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:47.884776    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zkfpq\""
Sep 17 17:52:47.896834 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:47.896815    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dc4ms"]
Sep 17 17:52:47.954335 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:47.954309    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fctg9\" (UniqueName: \"kubernetes.io/projected/e4521130-f0ac-4746-9d4d-9cd6dc1bd927-kube-api-access-fctg9\") pod \"authorino-7bcfd8f9cd-dc4ms\" (UID: \"e4521130-f0ac-4746-9d4d-9cd6dc1bd927\") " pod="kuadrant-system/authorino-7bcfd8f9cd-dc4ms"
Sep 17 17:52:48.055287 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:48.055231    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fctg9\" (UniqueName: \"kubernetes.io/projected/e4521130-f0ac-4746-9d4d-9cd6dc1bd927-kube-api-access-fctg9\") pod \"authorino-7bcfd8f9cd-dc4ms\" (UID: \"e4521130-f0ac-4746-9d4d-9cd6dc1bd927\") " pod="kuadrant-system/authorino-7bcfd8f9cd-dc4ms"
Sep 17 17:52:48.076025 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:48.076005    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6lknq"]
Sep 17 17:52:48.078538 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:52:48.078516    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c053a59_a277_4666_8612_aae4851ddfae.slice/crio-d49f9e11bc72aa1c1506f753f402993858aa4ecfeb7483f5d1a76420309836b2 WatchSource:0}: Error finding container d49f9e11bc72aa1c1506f753f402993858aa4ecfeb7483f5d1a76420309836b2: Status 404 returned error can't find the container with id d49f9e11bc72aa1c1506f753f402993858aa4ecfeb7483f5d1a76420309836b2
Sep 17 17:52:48.081595 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:48.081575    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fctg9\" (UniqueName: \"kubernetes.io/projected/e4521130-f0ac-4746-9d4d-9cd6dc1bd927-kube-api-access-fctg9\") pod \"authorino-7bcfd8f9cd-dc4ms\" (UID: \"e4521130-f0ac-4746-9d4d-9cd6dc1bd927\") " pod="kuadrant-system/authorino-7bcfd8f9cd-dc4ms"
Sep 17 17:52:48.191246 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:48.191208    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-dc4ms"
Sep 17 17:52:48.461181 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:48.460990    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dc4ms"]
Sep 17 17:52:48.463667 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:52:48.463627    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4521130_f0ac_4746_9d4d_9cd6dc1bd927.slice/crio-5603a7612d131e60afa1474e0abaaa92e22e1eb35fe38b71a81a0599af0ea4df WatchSource:0}: Error finding container 5603a7612d131e60afa1474e0abaaa92e22e1eb35fe38b71a81a0599af0ea4df: Status 404 returned error can't find the container with id 5603a7612d131e60afa1474e0abaaa92e22e1eb35fe38b71a81a0599af0ea4df
Sep 17 17:52:48.863099 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:48.863062    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-dc4ms" event={"ID":"e4521130-f0ac-4746-9d4d-9cd6dc1bd927","Type":"ContainerStarted","Data":"5603a7612d131e60afa1474e0abaaa92e22e1eb35fe38b71a81a0599af0ea4df"}
Sep 17 17:52:48.864208 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:48.864183    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-6lknq" event={"ID":"0c053a59-a277-4666-8612-aae4851ddfae","Type":"ContainerStarted","Data":"d49f9e11bc72aa1c1506f753f402993858aa4ecfeb7483f5d1a76420309836b2"}
Sep 17 17:52:49.869878 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:49.869843    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-6lknq" event={"ID":"0c053a59-a277-4666-8612-aae4851ddfae","Type":"ContainerStarted","Data":"773f7cfdea2770588ee80cbfb521db3b1f818d4b7f7708945a0bf14880e5b96c"}
Sep 17 17:52:49.870258 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:49.869923    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-6lknq"
Sep 17 17:52:49.890156 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:49.890103    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-6lknq" podStartSLOduration=1.540683391 podStartE2EDuration="2.890087735s" totalImagesPullingTime="1.349404344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:52:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:52:49.888260798 +0000 UTC m=+1080.979567582" watchObservedRunningTime="2026-09-17 17:52:49.890087735 +0000 UTC m=+1080.981394555"
Sep 17 17:52:51.308102 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:51.308024    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dc4ms"]
Sep 17 17:52:51.880565 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:51.880528    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-dc4ms" event={"ID":"e4521130-f0ac-4746-9d4d-9cd6dc1bd927","Type":"ContainerStarted","Data":"100dc4e081caa79f9d26a896247742346bc2404a45b5cc0d5cb452944dae0acc"}
Sep 17 17:52:51.899729 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:51.899652    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-dc4ms" podStartSLOduration=2.313021579 podStartE2EDuration="4.899635696s" totalImagesPullingTime="2.586614117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:52:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:52:51.898765498 +0000 UTC m=+1082.990072262" watchObservedRunningTime="2026-09-17 17:52:51.899635696 +0000 UTC m=+1082.990942461"
Sep 17 17:52:52.885056 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:52.885015    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-dc4ms" podUID="e4521130-f0ac-4746-9d4d-9cd6dc1bd927" containerName="authorino" containerID="cri-o://100dc4e081caa79f9d26a896247742346bc2404a45b5cc0d5cb452944dae0acc" gracePeriod=30
Sep 17 17:52:53.128588 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:53.128564    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-dc4ms"
Sep 17 17:52:53.198472 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:53.198406    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fctg9\" (UniqueName: \"kubernetes.io/projected/e4521130-f0ac-4746-9d4d-9cd6dc1bd927-kube-api-access-fctg9\") pod \"e4521130-f0ac-4746-9d4d-9cd6dc1bd927\" (UID: \"e4521130-f0ac-4746-9d4d-9cd6dc1bd927\") "
Sep 17 17:52:53.200503 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:53.200478    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4521130-f0ac-4746-9d4d-9cd6dc1bd927-kube-api-access-fctg9" (OuterVolumeSpecName: "kube-api-access-fctg9") pod "e4521130-f0ac-4746-9d4d-9cd6dc1bd927" (UID: "e4521130-f0ac-4746-9d4d-9cd6dc1bd927"). InnerVolumeSpecName "kube-api-access-fctg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:52:53.298905 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:53.298868    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fctg9\" (UniqueName: \"kubernetes.io/projected/e4521130-f0ac-4746-9d4d-9cd6dc1bd927-kube-api-access-fctg9\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 17:52:53.890420 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:53.890384    2583 generic.go:356] "Generic (PLEG): container finished" podID="e4521130-f0ac-4746-9d4d-9cd6dc1bd927" containerID="100dc4e081caa79f9d26a896247742346bc2404a45b5cc0d5cb452944dae0acc" exitCode=0
Sep 17 17:52:53.890420 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:53.890422    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-dc4ms"
Sep 17 17:52:53.890892 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:53.890476    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-dc4ms" event={"ID":"e4521130-f0ac-4746-9d4d-9cd6dc1bd927","Type":"ContainerDied","Data":"100dc4e081caa79f9d26a896247742346bc2404a45b5cc0d5cb452944dae0acc"}
Sep 17 17:52:53.890892 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:53.890518    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-dc4ms" event={"ID":"e4521130-f0ac-4746-9d4d-9cd6dc1bd927","Type":"ContainerDied","Data":"5603a7612d131e60afa1474e0abaaa92e22e1eb35fe38b71a81a0599af0ea4df"}
Sep 17 17:52:53.890892 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:53.890536    2583 scope.go:117] "RemoveContainer" containerID="100dc4e081caa79f9d26a896247742346bc2404a45b5cc0d5cb452944dae0acc"
Sep 17 17:52:53.898702 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:53.898686    2583 scope.go:117] "RemoveContainer" containerID="100dc4e081caa79f9d26a896247742346bc2404a45b5cc0d5cb452944dae0acc"
Sep 17 17:52:53.898963 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:52:53.898947    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"100dc4e081caa79f9d26a896247742346bc2404a45b5cc0d5cb452944dae0acc\": container with ID starting with 100dc4e081caa79f9d26a896247742346bc2404a45b5cc0d5cb452944dae0acc not found: ID does not exist" containerID="100dc4e081caa79f9d26a896247742346bc2404a45b5cc0d5cb452944dae0acc"
Sep 17 17:52:53.899011 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:53.898971    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"100dc4e081caa79f9d26a896247742346bc2404a45b5cc0d5cb452944dae0acc"} err="failed to get container status \"100dc4e081caa79f9d26a896247742346bc2404a45b5cc0d5cb452944dae0acc\": rpc error: code = NotFound desc = could not find container \"100dc4e081caa79f9d26a896247742346bc2404a45b5cc0d5cb452944dae0acc\": container with ID starting with 100dc4e081caa79f9d26a896247742346bc2404a45b5cc0d5cb452944dae0acc not found: ID does not exist"
Sep 17 17:52:53.914423 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:53.914393    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dc4ms"]
Sep 17 17:52:53.920535 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:53.920516    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dc4ms"]
Sep 17 17:52:55.040058 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:52:55.040027    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4521130-f0ac-4746-9d4d-9cd6dc1bd927" path="/var/lib/kubelet/pods/e4521130-f0ac-4746-9d4d-9cd6dc1bd927/volumes"
Sep 17 17:53:00.876602 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:53:00.876570    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-6lknq"
Sep 17 17:55:22.290989 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:55:22.290955    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-8v82t"]
Sep 17 17:55:22.291447 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:55:22.291262    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4521130-f0ac-4746-9d4d-9cd6dc1bd927" containerName="authorino"
Sep 17 17:55:22.291447 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:55:22.291275    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4521130-f0ac-4746-9d4d-9cd6dc1bd927" containerName="authorino"
Sep 17 17:55:22.291447 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:55:22.291329    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4521130-f0ac-4746-9d4d-9cd6dc1bd927" containerName="authorino"
Sep 17 17:55:22.294099 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:55:22.294083    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-8v82t"
Sep 17 17:55:22.296554 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:55:22.296530    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 17:55:22.296765 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:55:22.296733    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 17:55:22.296870 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:55:22.296827    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-65dq2\""
Sep 17 17:55:22.297497 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:55:22.297480    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 17:55:22.297497 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:55:22.297489    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 17:55:22.304305 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:55:22.304283    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-8v82t"]
Sep 17 17:55:22.342547 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:55:22.342521    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c860729e-690d-45bf-936f-a6c12e01415a-model-cache\") pod \"s3-init-8v82t\" (UID: \"c860729e-690d-45bf-936f-a6c12e01415a\") " pod="kserve/s3-init-8v82t"
Sep 17 17:55:22.342682 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:55:22.342564    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpd42\" (UniqueName: \"kubernetes.io/projected/c860729e-690d-45bf-936f-a6c12e01415a-kube-api-access-vpd42\") pod \"s3-init-8v82t\" (UID: \"c860729e-690d-45bf-936f-a6c12e01415a\") " pod="kserve/s3-init-8v82t"
Sep 17 17:55:22.443622 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:55:22.443597    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c860729e-690d-45bf-936f-a6c12e01415a-model-cache\") pod \"s3-init-8v82t\" (UID: \"c860729e-690d-45bf-936f-a6c12e01415a\") " pod="kserve/s3-init-8v82t"
Sep 17 17:55:22.443777 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:55:22.443643    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpd42\" (UniqueName: \"kubernetes.io/projected/c860729e-690d-45bf-936f-a6c12e01415a-kube-api-access-vpd42\") pod \"s3-init-8v82t\" (UID: \"c860729e-690d-45bf-936f-a6c12e01415a\") " pod="kserve/s3-init-8v82t"
Sep 17 17:55:22.443989 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:55:22.443971    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c860729e-690d-45bf-936f-a6c12e01415a-model-cache\") pod \"s3-init-8v82t\" (UID: \"c860729e-690d-45bf-936f-a6c12e01415a\") " pod="kserve/s3-init-8v82t"
Sep 17 17:55:22.452891 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:55:22.452870    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpd42\" (UniqueName: \"kubernetes.io/projected/c860729e-690d-45bf-936f-a6c12e01415a-kube-api-access-vpd42\") pod \"s3-init-8v82t\" (UID: \"c860729e-690d-45bf-936f-a6c12e01415a\") " pod="kserve/s3-init-8v82t"
Sep 17 17:55:22.602416 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:55:22.602392    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-8v82t"
Sep 17 17:55:22.876721 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:55:22.876628    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-8v82t"]
Sep 17 17:55:22.879568 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:55:22.879541    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc860729e_690d_45bf_936f_a6c12e01415a.slice/crio-8986cb7a1faec1190a4ff9d2633325968a6ce82646e1cc8b46b055c5b6e53ad0 WatchSource:0}: Error finding container 8986cb7a1faec1190a4ff9d2633325968a6ce82646e1cc8b46b055c5b6e53ad0: Status 404 returned error can't find the container with id 8986cb7a1faec1190a4ff9d2633325968a6ce82646e1cc8b46b055c5b6e53ad0
Sep 17 17:55:22.881415 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:55:22.881399    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:55:23.485317 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:55:23.485274    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8v82t" event={"ID":"c860729e-690d-45bf-936f-a6c12e01415a","Type":"ContainerStarted","Data":"8986cb7a1faec1190a4ff9d2633325968a6ce82646e1cc8b46b055c5b6e53ad0"}
Sep 17 17:55:27.508450 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:55:27.508411    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8v82t" event={"ID":"c860729e-690d-45bf-936f-a6c12e01415a","Type":"ContainerStarted","Data":"b4b395ffec064859d13ec21404675d70ffcee1c59fd0da58d81a294c9d0a0689"}
Sep 17 17:55:28.513860 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:55:28.513824    2583 generic.go:356] "Generic (PLEG): container finished" podID="c860729e-690d-45bf-936f-a6c12e01415a" containerID="b4b395ffec064859d13ec21404675d70ffcee1c59fd0da58d81a294c9d0a0689" exitCode=0
Sep 17 17:55:28.513860 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:55:28.513863    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8v82t" event={"ID":"c860729e-690d-45bf-936f-a6c12e01415a","Type":"ContainerDied","Data":"b4b395ffec064859d13ec21404675d70ffcee1c59fd0da58d81a294c9d0a0689"}
Sep 17 17:55:33.541237 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:55:33.541198    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8v82t" event={"ID":"c860729e-690d-45bf-936f-a6c12e01415a","Type":"ContainerStarted","Data":"fe3c175da0bec52ac062ad6422ea8cfab6b7a062f4760a0e2e83b468799f4873"}
Sep 17 17:55:33.561190 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:55:33.561137    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-8v82t" podStartSLOduration=1.882880143 podStartE2EDuration="11.561121539s" totalImagesPullingTime="7.912148984s" totalInitContainerRuntime="1.766092412s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:55:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:55:33.560182455 +0000 UTC m=+1244.651489251" watchObservedRunningTime="2026-09-17 17:55:33.561121539 +0000 UTC m=+1244.652428303"
Sep 17 17:56:38.811750 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:56:38.811714    2583 generic.go:356] "Generic (PLEG): container finished" podID="c860729e-690d-45bf-936f-a6c12e01415a" containerID="fe3c175da0bec52ac062ad6422ea8cfab6b7a062f4760a0e2e83b468799f4873" exitCode=0
Sep 17 17:56:38.812165 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:56:38.811759    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8v82t" event={"ID":"c860729e-690d-45bf-936f-a6c12e01415a","Type":"ContainerDied","Data":"fe3c175da0bec52ac062ad6422ea8cfab6b7a062f4760a0e2e83b468799f4873"}
Sep 17 17:56:39.934946 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:56:39.934920    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-8v82t"
Sep 17 17:56:40.071933 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:56:40.071846    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpd42\" (UniqueName: \"kubernetes.io/projected/c860729e-690d-45bf-936f-a6c12e01415a-kube-api-access-vpd42\") pod \"c860729e-690d-45bf-936f-a6c12e01415a\" (UID: \"c860729e-690d-45bf-936f-a6c12e01415a\") "
Sep 17 17:56:40.071933 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:56:40.071911    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c860729e-690d-45bf-936f-a6c12e01415a-model-cache\") pod \"c860729e-690d-45bf-936f-a6c12e01415a\" (UID: \"c860729e-690d-45bf-936f-a6c12e01415a\") "
Sep 17 17:56:40.074049 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:56:40.074018    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c860729e-690d-45bf-936f-a6c12e01415a-kube-api-access-vpd42" (OuterVolumeSpecName: "kube-api-access-vpd42") pod "c860729e-690d-45bf-936f-a6c12e01415a" (UID: "c860729e-690d-45bf-936f-a6c12e01415a"). InnerVolumeSpecName "kube-api-access-vpd42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:56:40.090631 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:56:40.090601    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c860729e-690d-45bf-936f-a6c12e01415a-model-cache" (OuterVolumeSpecName: "model-cache") pod "c860729e-690d-45bf-936f-a6c12e01415a" (UID: "c860729e-690d-45bf-936f-a6c12e01415a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:56:40.173362 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:56:40.173326    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpd42\" (UniqueName: \"kubernetes.io/projected/c860729e-690d-45bf-936f-a6c12e01415a-kube-api-access-vpd42\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 17:56:40.173362 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:56:40.173364    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c860729e-690d-45bf-936f-a6c12e01415a-model-cache\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 17:56:40.821641 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:56:40.821616    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-8v82t"
Sep 17 17:56:40.821641 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:56:40.821631    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8v82t" event={"ID":"c860729e-690d-45bf-936f-a6c12e01415a","Type":"ContainerDied","Data":"8986cb7a1faec1190a4ff9d2633325968a6ce82646e1cc8b46b055c5b6e53ad0"}
Sep 17 17:56:40.821862 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:56:40.821693    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8986cb7a1faec1190a4ff9d2633325968a6ce82646e1cc8b46b055c5b6e53ad0"
Sep 17 17:57:06.893799 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:57:06.893719    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb"]
Sep 17 17:57:06.894230 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:57:06.894011    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c860729e-690d-45bf-936f-a6c12e01415a" containerName="download-hf-model"
Sep 17 17:57:06.894230 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:57:06.894023    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c860729e-690d-45bf-936f-a6c12e01415a" containerName="download-hf-model"
Sep 17 17:57:06.894230 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:57:06.894037    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c860729e-690d-45bf-936f-a6c12e01415a" containerName="s3-init"
Sep 17 17:57:06.894230 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:57:06.894043    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c860729e-690d-45bf-936f-a6c12e01415a" containerName="s3-init"
Sep 17 17:57:06.894230 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:57:06.894092    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c860729e-690d-45bf-936f-a6c12e01415a" containerName="s3-init"
Sep 17 17:57:06.897138 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:57:06.897123    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb"
Sep 17 17:57:06.900476 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:57:06.900446    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-lxczf\""
Sep 17 17:57:06.900604 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:57:06.900498    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 17 17:57:06.901234 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:57:06.901216    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-h64nn\""
Sep 17 17:57:06.901348 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:57:06.901294    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 17 17:57:06.901432 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:57:06.901414    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 17 17:57:06.908305 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:57:06.908286    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb"]
Sep 17 17:57:06.972471 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:57:06.972445    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cf5ec29-eca3-4dfc-b630-97cbd1987919-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb\" (UID: \"9cf5ec29-eca3-4dfc-b630-97cbd1987919\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb"
Sep 17 17:57:06.972614 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:57:06.972589    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w6l4\" (UniqueName: \"kubernetes.io/projected/9cf5ec29-eca3-4dfc-b630-97cbd1987919-kube-api-access-4w6l4\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb\" (UID: \"9cf5ec29-eca3-4dfc-b630-97cbd1987919\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb"
Sep 17 17:57:07.073433 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:57:07.073405    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cf5ec29-eca3-4dfc-b630-97cbd1987919-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb\" (UID: \"9cf5ec29-eca3-4dfc-b630-97cbd1987919\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb"
Sep 17 17:57:07.073566 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:57:07.073457    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w6l4\" (UniqueName: \"kubernetes.io/projected/9cf5ec29-eca3-4dfc-b630-97cbd1987919-kube-api-access-4w6l4\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb\" (UID: \"9cf5ec29-eca3-4dfc-b630-97cbd1987919\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb"
Sep 17 17:57:07.075880 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:57:07.075850    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cf5ec29-eca3-4dfc-b630-97cbd1987919-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb\" (UID: \"9cf5ec29-eca3-4dfc-b630-97cbd1987919\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb"
Sep 17 17:57:07.082735 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:57:07.082712    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w6l4\" (UniqueName: \"kubernetes.io/projected/9cf5ec29-eca3-4dfc-b630-97cbd1987919-kube-api-access-4w6l4\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb\" (UID: \"9cf5ec29-eca3-4dfc-b630-97cbd1987919\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb"
Sep 17 17:57:07.206881 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:57:07.206819    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb"
Sep 17 17:57:07.486317 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:57:07.486240    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb"]
Sep 17 17:57:07.489469 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:57:07.489438    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cf5ec29_eca3_4dfc_b630_97cbd1987919.slice/crio-8d7135a06d8fc253cbf2d0e6787edcc21a7a187dac411d7ab51f27762c0d444b WatchSource:0}: Error finding container 8d7135a06d8fc253cbf2d0e6787edcc21a7a187dac411d7ab51f27762c0d444b: Status 404 returned error can't find the container with id 8d7135a06d8fc253cbf2d0e6787edcc21a7a187dac411d7ab51f27762c0d444b
Sep 17 17:57:07.933052 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:57:07.933017    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb" event={"ID":"9cf5ec29-eca3-4dfc-b630-97cbd1987919","Type":"ContainerStarted","Data":"8d7135a06d8fc253cbf2d0e6787edcc21a7a187dac411d7ab51f27762c0d444b"}
Sep 17 17:57:09.944313 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:57:09.944279    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb" event={"ID":"9cf5ec29-eca3-4dfc-b630-97cbd1987919","Type":"ContainerStarted","Data":"62b746a38544d8a668c8d82ee7214f01fb717b1947e188ca16f663bcba1eeee7"}
Sep 17 17:57:09.944705 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:57:09.944470    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb"
Sep 17 17:57:09.968068 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:57:09.968017    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb" podStartSLOduration=1.6992766910000001 podStartE2EDuration="3.968001306s" totalImagesPullingTime="2.268724615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:09.965190884 +0000 UTC m=+1341.056497648" watchObservedRunningTime="2026-09-17 17:57:09.968001306 +0000 UTC m=+1341.059308106"
Sep 17 17:57:40.950739 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:57:40.950712    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb"
Sep 17 17:57:51.218827 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:57:51.218796    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb"]
Sep 17 17:58:00.949890 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:58:00.949842    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb" podUID="9cf5ec29-eca3-4dfc-b630-97cbd1987919" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:58:02.092917 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:58:02.092853    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb" message="" err="command '/bin/sleep 15' exited with 137: "
Sep 17 17:58:02.092917 ip-10-0-141-30 kubenswrapper[2583]: E0917 17:58:02.092895    2583 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb" podUID="9cf5ec29-eca3-4dfc-b630-97cbd1987919" containerName="main" containerID="cri-o://62b746a38544d8a668c8d82ee7214f01fb717b1947e188ca16f663bcba1eeee7"
Sep 17 17:58:02.093274 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:58:02.092949    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb" podUID="9cf5ec29-eca3-4dfc-b630-97cbd1987919" containerName="main" containerID="cri-o://62b746a38544d8a668c8d82ee7214f01fb717b1947e188ca16f663bcba1eeee7" gracePeriod=50
Sep 17 17:58:02.156289 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:58:02.156265    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb_9cf5ec29-eca3-4dfc-b630-97cbd1987919/main/0.log"
Sep 17 17:58:02.156430 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:58:02.156312    2583 generic.go:356] "Generic (PLEG): container finished" podID="9cf5ec29-eca3-4dfc-b630-97cbd1987919" containerID="62b746a38544d8a668c8d82ee7214f01fb717b1947e188ca16f663bcba1eeee7" exitCode=1
Sep 17 17:58:02.156430 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:58:02.156397    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb" event={"ID":"9cf5ec29-eca3-4dfc-b630-97cbd1987919","Type":"ContainerDied","Data":"62b746a38544d8a668c8d82ee7214f01fb717b1947e188ca16f663bcba1eeee7"}
Sep 17 17:58:02.229509 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:58:02.229485    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb_9cf5ec29-eca3-4dfc-b630-97cbd1987919/main/0.log"
Sep 17 17:58:02.231653 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:58:02.231637    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb"
Sep 17 17:58:02.379889 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:58:02.379860    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w6l4\" (UniqueName: \"kubernetes.io/projected/9cf5ec29-eca3-4dfc-b630-97cbd1987919-kube-api-access-4w6l4\") pod \"9cf5ec29-eca3-4dfc-b630-97cbd1987919\" (UID: \"9cf5ec29-eca3-4dfc-b630-97cbd1987919\") "
Sep 17 17:58:02.380086 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:58:02.379920    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cf5ec29-eca3-4dfc-b630-97cbd1987919-tls-certs\") pod \"9cf5ec29-eca3-4dfc-b630-97cbd1987919\" (UID: \"9cf5ec29-eca3-4dfc-b630-97cbd1987919\") "
Sep 17 17:58:02.382083 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:58:02.382055    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cf5ec29-eca3-4dfc-b630-97cbd1987919-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9cf5ec29-eca3-4dfc-b630-97cbd1987919" (UID: "9cf5ec29-eca3-4dfc-b630-97cbd1987919"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:58:02.382083 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:58:02.382055    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cf5ec29-eca3-4dfc-b630-97cbd1987919-kube-api-access-4w6l4" (OuterVolumeSpecName: "kube-api-access-4w6l4") pod "9cf5ec29-eca3-4dfc-b630-97cbd1987919" (UID: "9cf5ec29-eca3-4dfc-b630-97cbd1987919"). InnerVolumeSpecName "kube-api-access-4w6l4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:58:02.480841 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:58:02.480802    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cf5ec29-eca3-4dfc-b630-97cbd1987919-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 17:58:02.480841 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:58:02.480831    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w6l4\" (UniqueName: \"kubernetes.io/projected/9cf5ec29-eca3-4dfc-b630-97cbd1987919-kube-api-access-4w6l4\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 17:58:03.162023 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:58:03.161996    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb_9cf5ec29-eca3-4dfc-b630-97cbd1987919/main/0.log"
Sep 17 17:58:03.162414 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:58:03.162098    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb"
Sep 17 17:58:03.162414 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:58:03.162138    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb" event={"ID":"9cf5ec29-eca3-4dfc-b630-97cbd1987919","Type":"ContainerDied","Data":"8d7135a06d8fc253cbf2d0e6787edcc21a7a187dac411d7ab51f27762c0d444b"}
Sep 17 17:58:03.162414 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:58:03.162198    2583 scope.go:117] "RemoveContainer" containerID="62b746a38544d8a668c8d82ee7214f01fb717b1947e188ca16f663bcba1eeee7"
Sep 17 17:58:03.180448 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:58:03.180426    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb"]
Sep 17 17:58:03.184357 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:58:03.184335    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5mghfb"]
Sep 17 17:58:05.039902 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:58:05.039873    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cf5ec29-eca3-4dfc-b630-97cbd1987919" path="/var/lib/kubelet/pods/9cf5ec29-eca3-4dfc-b630-97cbd1987919/volumes"
Sep 17 17:59:02.460366 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:02.460321    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts"]
Sep 17 17:59:02.460975 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:02.460898    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cf5ec29-eca3-4dfc-b630-97cbd1987919" containerName="main"
Sep 17 17:59:02.460975 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:02.460918    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cf5ec29-eca3-4dfc-b630-97cbd1987919" containerName="main"
Sep 17 17:59:02.461083 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:02.461031    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cf5ec29-eca3-4dfc-b630-97cbd1987919" containerName="main"
Sep 17 17:59:02.465266 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:02.465246    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts"
Sep 17 17:59:02.467750 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:02.467728    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 17 17:59:02.467867 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:02.467802    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-6n7g9\""
Sep 17 17:59:02.468632 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:02.468610    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 17 17:59:02.468749 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:02.468692    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 17 17:59:02.468812 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:02.468696    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-nsg5j\""
Sep 17 17:59:02.476107 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:02.476084    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts"]
Sep 17 17:59:02.512618 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:02.512587    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hn82\" (UniqueName: \"kubernetes.io/projected/b802e985-4818-47ab-956a-d0e2bcc5bb4f-kube-api-access-4hn82\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts\" (UID: \"b802e985-4818-47ab-956a-d0e2bcc5bb4f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts"
Sep 17 17:59:02.512795 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:02.512626    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b802e985-4818-47ab-956a-d0e2bcc5bb4f-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts\" (UID: \"b802e985-4818-47ab-956a-d0e2bcc5bb4f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts"
Sep 17 17:59:02.512795 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:02.512644    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b802e985-4818-47ab-956a-d0e2bcc5bb4f-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts\" (UID: \"b802e985-4818-47ab-956a-d0e2bcc5bb4f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts"
Sep 17 17:59:02.512795 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:02.512707    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/b802e985-4818-47ab-956a-d0e2bcc5bb4f-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts\" (UID: \"b802e985-4818-47ab-956a-d0e2bcc5bb4f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts"
Sep 17 17:59:02.613561 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:02.613526    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hn82\" (UniqueName: \"kubernetes.io/projected/b802e985-4818-47ab-956a-d0e2bcc5bb4f-kube-api-access-4hn82\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts\" (UID: \"b802e985-4818-47ab-956a-d0e2bcc5bb4f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts"
Sep 17 17:59:02.613810 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:02.613565    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b802e985-4818-47ab-956a-d0e2bcc5bb4f-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts\" (UID: \"b802e985-4818-47ab-956a-d0e2bcc5bb4f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts"
Sep 17 17:59:02.613810 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:02.613694    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b802e985-4818-47ab-956a-d0e2bcc5bb4f-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts\" (UID: \"b802e985-4818-47ab-956a-d0e2bcc5bb4f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts"
Sep 17 17:59:02.613810 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:02.613737    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/b802e985-4818-47ab-956a-d0e2bcc5bb4f-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts\" (UID: \"b802e985-4818-47ab-956a-d0e2bcc5bb4f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts"
Sep 17 17:59:02.614052 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:02.614033    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b802e985-4818-47ab-956a-d0e2bcc5bb4f-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts\" (UID: \"b802e985-4818-47ab-956a-d0e2bcc5bb4f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts"
Sep 17 17:59:02.614114 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:02.614059    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/b802e985-4818-47ab-956a-d0e2bcc5bb4f-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts\" (UID: \"b802e985-4818-47ab-956a-d0e2bcc5bb4f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts"
Sep 17 17:59:02.616055 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:02.616036    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b802e985-4818-47ab-956a-d0e2bcc5bb4f-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts\" (UID: \"b802e985-4818-47ab-956a-d0e2bcc5bb4f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts"
Sep 17 17:59:02.623048 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:02.623014    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hn82\" (UniqueName: \"kubernetes.io/projected/b802e985-4818-47ab-956a-d0e2bcc5bb4f-kube-api-access-4hn82\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts\" (UID: \"b802e985-4818-47ab-956a-d0e2bcc5bb4f\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts"
Sep 17 17:59:02.778641 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:02.778538    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts"
Sep 17 17:59:03.062531 ip-10-0-141-30 kubenswrapper[2583]: W0917 17:59:03.062500    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb802e985_4818_47ab_956a_d0e2bcc5bb4f.slice/crio-7ff297f8a882bca5b87c06ffbc010b4d3f2aa99a1109928c02092b7f4ac05f9a WatchSource:0}: Error finding container 7ff297f8a882bca5b87c06ffbc010b4d3f2aa99a1109928c02092b7f4ac05f9a: Status 404 returned error can't find the container with id 7ff297f8a882bca5b87c06ffbc010b4d3f2aa99a1109928c02092b7f4ac05f9a
Sep 17 17:59:03.401982 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:03.401952    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts" event={"ID":"b802e985-4818-47ab-956a-d0e2bcc5bb4f","Type":"ContainerStarted","Data":"1a27af1d6c9fbc5ee053d4eb7f71ea4d6804543b57517db9454dc1795fec48b2"}
Sep 17 17:59:03.402157 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:03.402001    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts" event={"ID":"b802e985-4818-47ab-956a-d0e2bcc5bb4f","Type":"ContainerStarted","Data":"7ff297f8a882bca5b87c06ffbc010b4d3f2aa99a1109928c02092b7f4ac05f9a"}
Sep 17 17:59:03.402157 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:03.402051    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts"
Sep 17 17:59:03.633494 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:03.633463    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts"]
Sep 17 17:59:03.652831 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:03.652751    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts" podStartSLOduration=1.6527377250000002 podStartE2EDuration="1.652737725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:59:03.651202582 +0000 UTC m=+1454.742509360" watchObservedRunningTime="2026-09-17 17:59:03.652737725 +0000 UTC m=+1454.744044489"
Sep 17 17:59:35.412495 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:35.412464    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts"
Sep 17 17:59:39.428192 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:39.428157    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts"]
Sep 17 17:59:45.412221 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:45.412179    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts" podUID="b802e985-4818-47ab-956a-d0e2bcc5bb4f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 17:59:54.448804 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:54.448745    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts" podUID="b802e985-4818-47ab-956a-d0e2bcc5bb4f" containerName="main" containerID="cri-o://1a27af1d6c9fbc5ee053d4eb7f71ea4d6804543b57517db9454dc1795fec48b2" gracePeriod=45
Sep 17 17:59:55.412174 ip-10-0-141-30 kubenswrapper[2583]: I0917 17:59:55.412134    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts" podUID="b802e985-4818-47ab-956a-d0e2bcc5bb4f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:00:05.413058 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:05.412975    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts" podUID="b802e985-4818-47ab-956a-d0e2bcc5bb4f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:00:05.413579 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:05.413081    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts"
Sep 17 18:00:15.411970 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:15.411927    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts" podUID="b802e985-4818-47ab-956a-d0e2bcc5bb4f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:00:24.687126 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:24.687102    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts"
Sep 17 18:00:24.735401 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:24.735316    2583 generic.go:356] "Generic (PLEG): container finished" podID="b802e985-4818-47ab-956a-d0e2bcc5bb4f" containerID="1a27af1d6c9fbc5ee053d4eb7f71ea4d6804543b57517db9454dc1795fec48b2" exitCode=0
Sep 17 18:00:24.735401 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:24.735372    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts"
Sep 17 18:00:24.735596 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:24.735402    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts" event={"ID":"b802e985-4818-47ab-956a-d0e2bcc5bb4f","Type":"ContainerDied","Data":"1a27af1d6c9fbc5ee053d4eb7f71ea4d6804543b57517db9454dc1795fec48b2"}
Sep 17 18:00:24.735596 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:24.735457    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts" event={"ID":"b802e985-4818-47ab-956a-d0e2bcc5bb4f","Type":"ContainerDied","Data":"7ff297f8a882bca5b87c06ffbc010b4d3f2aa99a1109928c02092b7f4ac05f9a"}
Sep 17 18:00:24.735596 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:24.735479    2583 scope.go:117] "RemoveContainer" containerID="1a27af1d6c9fbc5ee053d4eb7f71ea4d6804543b57517db9454dc1795fec48b2"
Sep 17 18:00:24.736497 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:24.736475    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b802e985-4818-47ab-956a-d0e2bcc5bb4f-tokenizer-cache\") pod \"b802e985-4818-47ab-956a-d0e2bcc5bb4f\" (UID: \"b802e985-4818-47ab-956a-d0e2bcc5bb4f\") "
Sep 17 18:00:24.736549 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:24.736518    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b802e985-4818-47ab-956a-d0e2bcc5bb4f-tls-certs\") pod \"b802e985-4818-47ab-956a-d0e2bcc5bb4f\" (UID: \"b802e985-4818-47ab-956a-d0e2bcc5bb4f\") "
Sep 17 18:00:24.736603 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:24.736551    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/b802e985-4818-47ab-956a-d0e2bcc5bb4f-cachi2-cache\") pod \"b802e985-4818-47ab-956a-d0e2bcc5bb4f\" (UID: \"b802e985-4818-47ab-956a-d0e2bcc5bb4f\") "
Sep 17 18:00:24.736681 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:24.736632    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hn82\" (UniqueName: \"kubernetes.io/projected/b802e985-4818-47ab-956a-d0e2bcc5bb4f-kube-api-access-4hn82\") pod \"b802e985-4818-47ab-956a-d0e2bcc5bb4f\" (UID: \"b802e985-4818-47ab-956a-d0e2bcc5bb4f\") "
Sep 17 18:00:24.736818 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:24.736796    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b802e985-4818-47ab-956a-d0e2bcc5bb4f-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "b802e985-4818-47ab-956a-d0e2bcc5bb4f" (UID: "b802e985-4818-47ab-956a-d0e2bcc5bb4f"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:24.736908 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:24.736889    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b802e985-4818-47ab-956a-d0e2bcc5bb4f-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "b802e985-4818-47ab-956a-d0e2bcc5bb4f" (UID: "b802e985-4818-47ab-956a-d0e2bcc5bb4f"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:24.736964 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:24.736952    2583 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b802e985-4818-47ab-956a-d0e2bcc5bb4f-tokenizer-cache\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:00:24.737015 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:24.736966    2583 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/b802e985-4818-47ab-956a-d0e2bcc5bb4f-cachi2-cache\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:00:24.738887 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:24.738855    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b802e985-4818-47ab-956a-d0e2bcc5bb4f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b802e985-4818-47ab-956a-d0e2bcc5bb4f" (UID: "b802e985-4818-47ab-956a-d0e2bcc5bb4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:24.738961 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:24.738909    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b802e985-4818-47ab-956a-d0e2bcc5bb4f-kube-api-access-4hn82" (OuterVolumeSpecName: "kube-api-access-4hn82") pod "b802e985-4818-47ab-956a-d0e2bcc5bb4f" (UID: "b802e985-4818-47ab-956a-d0e2bcc5bb4f"). InnerVolumeSpecName "kube-api-access-4hn82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:00:24.753016 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:24.752991    2583 scope.go:117] "RemoveContainer" containerID="1a27af1d6c9fbc5ee053d4eb7f71ea4d6804543b57517db9454dc1795fec48b2"
Sep 17 18:00:24.753296 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:00:24.753275    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a27af1d6c9fbc5ee053d4eb7f71ea4d6804543b57517db9454dc1795fec48b2\": container with ID starting with 1a27af1d6c9fbc5ee053d4eb7f71ea4d6804543b57517db9454dc1795fec48b2 not found: ID does not exist" containerID="1a27af1d6c9fbc5ee053d4eb7f71ea4d6804543b57517db9454dc1795fec48b2"
Sep 17 18:00:24.753345 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:24.753305    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a27af1d6c9fbc5ee053d4eb7f71ea4d6804543b57517db9454dc1795fec48b2"} err="failed to get container status \"1a27af1d6c9fbc5ee053d4eb7f71ea4d6804543b57517db9454dc1795fec48b2\": rpc error: code = NotFound desc = could not find container \"1a27af1d6c9fbc5ee053d4eb7f71ea4d6804543b57517db9454dc1795fec48b2\": container with ID starting with 1a27af1d6c9fbc5ee053d4eb7f71ea4d6804543b57517db9454dc1795fec48b2 not found: ID does not exist"
Sep 17 18:00:24.837903 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:24.837876    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hn82\" (UniqueName: \"kubernetes.io/projected/b802e985-4818-47ab-956a-d0e2bcc5bb4f-kube-api-access-4hn82\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:00:24.837903 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:24.837901    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b802e985-4818-47ab-956a-d0e2bcc5bb4f-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:00:25.057655 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:25.057628    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts"]
Sep 17 18:00:25.061925 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:25.061905    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcj42ts"]
Sep 17 18:00:27.040212 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:27.040183    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b802e985-4818-47ab-956a-d0e2bcc5bb4f" path="/var/lib/kubelet/pods/b802e985-4818-47ab-956a-d0e2bcc5bb4f/volumes"
Sep 17 18:00:40.145387 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.145348    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"]
Sep 17 18:00:40.147686 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.145787    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b802e985-4818-47ab-956a-d0e2bcc5bb4f" containerName="main"
Sep 17 18:00:40.147686 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.145803    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b802e985-4818-47ab-956a-d0e2bcc5bb4f" containerName="main"
Sep 17 18:00:40.147686 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.145887    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b802e985-4818-47ab-956a-d0e2bcc5bb4f" containerName="main"
Sep 17 18:00:40.148941 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.148923    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"
Sep 17 18:00:40.151498 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.151476    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 17 18:00:40.152357 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.152334    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-7smbt\""
Sep 17 18:00:40.152456 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.152409    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 17 18:00:40.152456 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.152424    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 17 18:00:40.152571 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.152423    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-p6528\""
Sep 17 18:00:40.163058 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.163036    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"]
Sep 17 18:00:40.255913 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.255879    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac5fbe45-640c-4550-b751-05153ab3a86f-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw\" (UID: \"ac5fbe45-640c-4550-b751-05153ab3a86f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"
Sep 17 18:00:40.256072 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.255918    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac5fbe45-640c-4550-b751-05153ab3a86f-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw\" (UID: \"ac5fbe45-640c-4550-b751-05153ab3a86f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"
Sep 17 18:00:40.256072 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.255943    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac5fbe45-640c-4550-b751-05153ab3a86f-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw\" (UID: \"ac5fbe45-640c-4550-b751-05153ab3a86f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"
Sep 17 18:00:40.256072 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.255987    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8vd5\" (UniqueName: \"kubernetes.io/projected/ac5fbe45-640c-4550-b751-05153ab3a86f-kube-api-access-q8vd5\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw\" (UID: \"ac5fbe45-640c-4550-b751-05153ab3a86f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"
Sep 17 18:00:40.256072 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.256017    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac5fbe45-640c-4550-b751-05153ab3a86f-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw\" (UID: \"ac5fbe45-640c-4550-b751-05153ab3a86f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"
Sep 17 18:00:40.256214 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.256072    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac5fbe45-640c-4550-b751-05153ab3a86f-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw\" (UID: \"ac5fbe45-640c-4550-b751-05153ab3a86f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"
Sep 17 18:00:40.357268 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.357236    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8vd5\" (UniqueName: \"kubernetes.io/projected/ac5fbe45-640c-4550-b751-05153ab3a86f-kube-api-access-q8vd5\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw\" (UID: \"ac5fbe45-640c-4550-b751-05153ab3a86f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"
Sep 17 18:00:40.357442 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.357284    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac5fbe45-640c-4550-b751-05153ab3a86f-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw\" (UID: \"ac5fbe45-640c-4550-b751-05153ab3a86f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"
Sep 17 18:00:40.357442 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.357319    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac5fbe45-640c-4550-b751-05153ab3a86f-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw\" (UID: \"ac5fbe45-640c-4550-b751-05153ab3a86f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"
Sep 17 18:00:40.357442 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.357367    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac5fbe45-640c-4550-b751-05153ab3a86f-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw\" (UID: \"ac5fbe45-640c-4550-b751-05153ab3a86f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"
Sep 17 18:00:40.357442 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.357393    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac5fbe45-640c-4550-b751-05153ab3a86f-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw\" (UID: \"ac5fbe45-640c-4550-b751-05153ab3a86f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"
Sep 17 18:00:40.357685 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.357636    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac5fbe45-640c-4550-b751-05153ab3a86f-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw\" (UID: \"ac5fbe45-640c-4550-b751-05153ab3a86f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"
Sep 17 18:00:40.357807 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.357789    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac5fbe45-640c-4550-b751-05153ab3a86f-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw\" (UID: \"ac5fbe45-640c-4550-b751-05153ab3a86f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"
Sep 17 18:00:40.357867 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.357846    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac5fbe45-640c-4550-b751-05153ab3a86f-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw\" (UID: \"ac5fbe45-640c-4550-b751-05153ab3a86f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"
Sep 17 18:00:40.357954 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.357936    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac5fbe45-640c-4550-b751-05153ab3a86f-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw\" (UID: \"ac5fbe45-640c-4550-b751-05153ab3a86f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"
Sep 17 18:00:40.359862 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.359837    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac5fbe45-640c-4550-b751-05153ab3a86f-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw\" (UID: \"ac5fbe45-640c-4550-b751-05153ab3a86f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"
Sep 17 18:00:40.360464 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.360442    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac5fbe45-640c-4550-b751-05153ab3a86f-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw\" (UID: \"ac5fbe45-640c-4550-b751-05153ab3a86f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"
Sep 17 18:00:40.368718 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.368695    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8vd5\" (UniqueName: \"kubernetes.io/projected/ac5fbe45-640c-4550-b751-05153ab3a86f-kube-api-access-q8vd5\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw\" (UID: \"ac5fbe45-640c-4550-b751-05153ab3a86f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"
Sep 17 18:00:40.371112 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.371090    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl"]
Sep 17 18:00:40.374630 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.374607    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl"
Sep 17 18:00:40.377006 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.376988    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-5rxfn\""
Sep 17 18:00:40.383209 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.383189    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl"]
Sep 17 18:00:40.458349 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.458273    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"
Sep 17 18:00:40.458478 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.458425    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8w98\" (UniqueName: \"kubernetes.io/projected/81fcef74-838d-4177-9c15-aa421058f9cc-kube-api-access-s8w98\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl\" (UID: \"81fcef74-838d-4177-9c15-aa421058f9cc\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl"
Sep 17 18:00:40.458478 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.458457    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81fcef74-838d-4177-9c15-aa421058f9cc-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl\" (UID: \"81fcef74-838d-4177-9c15-aa421058f9cc\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl"
Sep 17 18:00:40.559747 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.559714    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8w98\" (UniqueName: \"kubernetes.io/projected/81fcef74-838d-4177-9c15-aa421058f9cc-kube-api-access-s8w98\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl\" (UID: \"81fcef74-838d-4177-9c15-aa421058f9cc\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl"
Sep 17 18:00:40.559885 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.559754    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81fcef74-838d-4177-9c15-aa421058f9cc-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl\" (UID: \"81fcef74-838d-4177-9c15-aa421058f9cc\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl"
Sep 17 18:00:40.562228 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.562204    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81fcef74-838d-4177-9c15-aa421058f9cc-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl\" (UID: \"81fcef74-838d-4177-9c15-aa421058f9cc\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl"
Sep 17 18:00:40.569926 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.569905    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8w98\" (UniqueName: \"kubernetes.io/projected/81fcef74-838d-4177-9c15-aa421058f9cc-kube-api-access-s8w98\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl\" (UID: \"81fcef74-838d-4177-9c15-aa421058f9cc\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl"
Sep 17 18:00:40.689019 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.688992    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl"
Sep 17 18:00:40.735927 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.735842    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"]
Sep 17 18:00:40.744116 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.743583    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:00:40.801031 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.800999    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw" event={"ID":"ac5fbe45-640c-4550-b751-05153ab3a86f","Type":"ContainerStarted","Data":"81f429d41389b4a0221aa273c278ba8046765d6750cd30cb1a429d3074858411"}
Sep 17 18:00:40.961089 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:40.961067    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl"]
Sep 17 18:00:40.963574 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:00:40.963551    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81fcef74_838d_4177_9c15_aa421058f9cc.slice/crio-552dc6ed979cb7b9e64d723e47c82cc07a68d4b337c9ed95f0fe4d152aec407e WatchSource:0}: Error finding container 552dc6ed979cb7b9e64d723e47c82cc07a68d4b337c9ed95f0fe4d152aec407e: Status 404 returned error can't find the container with id 552dc6ed979cb7b9e64d723e47c82cc07a68d4b337c9ed95f0fe4d152aec407e
Sep 17 18:00:41.807179 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:41.807142    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl" event={"ID":"81fcef74-838d-4177-9c15-aa421058f9cc","Type":"ContainerStarted","Data":"5df20b913f9491b25c7cb3f590fbcbd73112dbc65870cad6a4a7ccc802af363e"}
Sep 17 18:00:41.807568 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:41.807196    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl" event={"ID":"81fcef74-838d-4177-9c15-aa421058f9cc","Type":"ContainerStarted","Data":"552dc6ed979cb7b9e64d723e47c82cc07a68d4b337c9ed95f0fe4d152aec407e"}
Sep 17 18:00:41.807568 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:41.807279    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl"
Sep 17 18:00:41.825745 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:41.825699    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl" podStartSLOduration=1.825684794 podStartE2EDuration="1.825684794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:00:41.824490603 +0000 UTC m=+1552.915797380" watchObservedRunningTime="2026-09-17 18:00:41.825684794 +0000 UTC m=+1552.916991553"
Sep 17 18:00:42.813032 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:42.812945    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw" event={"ID":"ac5fbe45-640c-4550-b751-05153ab3a86f","Type":"ContainerStarted","Data":"fcfd10dc2334767448ed6304762c44a3f6c66ebd6d66e12990fc5db248b77c5b"}
Sep 17 18:00:42.813404 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:42.813258    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"
Sep 17 18:00:44.824110 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:44.824073    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw" event={"ID":"ac5fbe45-640c-4550-b751-05153ab3a86f","Type":"ContainerStarted","Data":"d46807755338e662a2e8bfefdddceb52c8c9955d82ccafdd88fa6c0b6a27c6d7"}
Sep 17 18:00:44.845279 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:44.845234    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw" podStartSLOduration=1.6462438050000001 podStartE2EDuration="4.845219002s" totalImagesPullingTime="3.198975197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:00:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:00:44.844016214 +0000 UTC m=+1555.935322989" watchObservedRunningTime="2026-09-17 18:00:44.845219002 +0000 UTC m=+1555.936525767"
Sep 17 18:00:45.458939 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:45.458907    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"
Sep 17 18:00:50.460453 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:50.460419    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"
Sep 17 18:00:50.472843 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:50.472816    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"
Sep 17 18:00:50.864287 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:00:50.864256    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"
Sep 17 18:01:01.870598 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:01:01.870566    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"
Sep 17 18:01:12.814512 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:01:12.814483    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl"
Sep 17 18:02:26.663914 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:26.663883    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"]
Sep 17 18:02:26.670198 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:26.670170    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl"]
Sep 17 18:02:26.674236 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:02:26.674204    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:02:26.674236 ip-10-0-141-30 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:02:26.674236 ip-10-0-141-30 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 18:02:26.674236 ip-10-0-141-30 kubenswrapper[2583]:  > containerID="d46807755338e662a2e8bfefdddceb52c8c9955d82ccafdd88fa6c0b6a27c6d7" cmd=["/bin/sleep","15"]
Sep 17 18:02:26.674528 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:26.674254    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw" message="" err=<
Sep 17 18:02:26.674528 ip-10-0-141-30 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:02:26.674528 ip-10-0-141-30 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 18:02:26.674528 ip-10-0-141-30 kubenswrapper[2583]:  >
Sep 17 18:02:26.674528 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:02:26.674266    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:02:26.674528 ip-10-0-141-30 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:02:26.674528 ip-10-0-141-30 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 18:02:26.674528 ip-10-0-141-30 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw" podUID="ac5fbe45-640c-4550-b751-05153ab3a86f" containerName="main" containerID="cri-o://d46807755338e662a2e8bfefdddceb52c8c9955d82ccafdd88fa6c0b6a27c6d7"
Sep 17 18:02:26.674528 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:26.674311    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw" podUID="ac5fbe45-640c-4550-b751-05153ab3a86f" containerName="main" containerID="cri-o://d46807755338e662a2e8bfefdddceb52c8c9955d82ccafdd88fa6c0b6a27c6d7" gracePeriod=60
Sep 17 18:02:26.854277 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:26.854239    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw" podUID="ac5fbe45-640c-4550-b751-05153ab3a86f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8001/health\": dial tcp 10.134.0.28:8001: connect: connection refused"
Sep 17 18:02:26.890411 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:26.890382    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw" podUID="ac5fbe45-640c-4550-b751-05153ab3a86f" containerName="llm-d-routing-sidecar" containerID="cri-o://fcfd10dc2334767448ed6304762c44a3f6c66ebd6d66e12990fc5db248b77c5b" gracePeriod=60
Sep 17 18:02:27.140921 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.140894    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"
Sep 17 18:02:27.223892 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.223865    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac5fbe45-640c-4550-b751-05153ab3a86f-model-cache\") pod \"ac5fbe45-640c-4550-b751-05153ab3a86f\" (UID: \"ac5fbe45-640c-4550-b751-05153ab3a86f\") "
Sep 17 18:02:27.224043 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.223926    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac5fbe45-640c-4550-b751-05153ab3a86f-home\") pod \"ac5fbe45-640c-4550-b751-05153ab3a86f\" (UID: \"ac5fbe45-640c-4550-b751-05153ab3a86f\") "
Sep 17 18:02:27.224043 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.223949    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac5fbe45-640c-4550-b751-05153ab3a86f-tls-certs\") pod \"ac5fbe45-640c-4550-b751-05153ab3a86f\" (UID: \"ac5fbe45-640c-4550-b751-05153ab3a86f\") "
Sep 17 18:02:27.224043 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.223971    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac5fbe45-640c-4550-b751-05153ab3a86f-tmp-dir\") pod \"ac5fbe45-640c-4550-b751-05153ab3a86f\" (UID: \"ac5fbe45-640c-4550-b751-05153ab3a86f\") "
Sep 17 18:02:27.224043 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.223992    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8vd5\" (UniqueName: \"kubernetes.io/projected/ac5fbe45-640c-4550-b751-05153ab3a86f-kube-api-access-q8vd5\") pod \"ac5fbe45-640c-4550-b751-05153ab3a86f\" (UID: \"ac5fbe45-640c-4550-b751-05153ab3a86f\") "
Sep 17 18:02:27.224043 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.224026    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac5fbe45-640c-4550-b751-05153ab3a86f-dshm\") pod \"ac5fbe45-640c-4550-b751-05153ab3a86f\" (UID: \"ac5fbe45-640c-4550-b751-05153ab3a86f\") "
Sep 17 18:02:27.224292 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.224166    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac5fbe45-640c-4550-b751-05153ab3a86f-model-cache" (OuterVolumeSpecName: "model-cache") pod "ac5fbe45-640c-4550-b751-05153ab3a86f" (UID: "ac5fbe45-640c-4550-b751-05153ab3a86f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:27.224292 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.224256    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac5fbe45-640c-4550-b751-05153ab3a86f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ac5fbe45-640c-4550-b751-05153ab3a86f" (UID: "ac5fbe45-640c-4550-b751-05153ab3a86f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:27.224371 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.224306    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac5fbe45-640c-4550-b751-05153ab3a86f-home" (OuterVolumeSpecName: "home") pod "ac5fbe45-640c-4550-b751-05153ab3a86f" (UID: "ac5fbe45-640c-4550-b751-05153ab3a86f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:27.226261 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.226229    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac5fbe45-640c-4550-b751-05153ab3a86f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ac5fbe45-640c-4550-b751-05153ab3a86f" (UID: "ac5fbe45-640c-4550-b751-05153ab3a86f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:02:27.226261 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.226239    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac5fbe45-640c-4550-b751-05153ab3a86f-dshm" (OuterVolumeSpecName: "dshm") pod "ac5fbe45-640c-4550-b751-05153ab3a86f" (UID: "ac5fbe45-640c-4550-b751-05153ab3a86f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:27.226396 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.226251    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac5fbe45-640c-4550-b751-05153ab3a86f-kube-api-access-q8vd5" (OuterVolumeSpecName: "kube-api-access-q8vd5") pod "ac5fbe45-640c-4550-b751-05153ab3a86f" (UID: "ac5fbe45-640c-4550-b751-05153ab3a86f"). InnerVolumeSpecName "kube-api-access-q8vd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:02:27.271019 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.270961    2583 generic.go:356] "Generic (PLEG): container finished" podID="ac5fbe45-640c-4550-b751-05153ab3a86f" containerID="d46807755338e662a2e8bfefdddceb52c8c9955d82ccafdd88fa6c0b6a27c6d7" exitCode=0
Sep 17 18:02:27.271019 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.270987    2583 generic.go:356] "Generic (PLEG): container finished" podID="ac5fbe45-640c-4550-b751-05153ab3a86f" containerID="fcfd10dc2334767448ed6304762c44a3f6c66ebd6d66e12990fc5db248b77c5b" exitCode=0
Sep 17 18:02:27.271019 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.271011    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw" event={"ID":"ac5fbe45-640c-4550-b751-05153ab3a86f","Type":"ContainerDied","Data":"d46807755338e662a2e8bfefdddceb52c8c9955d82ccafdd88fa6c0b6a27c6d7"}
Sep 17 18:02:27.271019 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.271020    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"
Sep 17 18:02:27.271211 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.271066    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw" event={"ID":"ac5fbe45-640c-4550-b751-05153ab3a86f","Type":"ContainerDied","Data":"fcfd10dc2334767448ed6304762c44a3f6c66ebd6d66e12990fc5db248b77c5b"}
Sep 17 18:02:27.271211 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.271090    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw" event={"ID":"ac5fbe45-640c-4550-b751-05153ab3a86f","Type":"ContainerDied","Data":"81f429d41389b4a0221aa273c278ba8046765d6750cd30cb1a429d3074858411"}
Sep 17 18:02:27.271211 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.271119    2583 scope.go:117] "RemoveContainer" containerID="d46807755338e662a2e8bfefdddceb52c8c9955d82ccafdd88fa6c0b6a27c6d7"
Sep 17 18:02:27.279489 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.279469    2583 scope.go:117] "RemoveContainer" containerID="fcfd10dc2334767448ed6304762c44a3f6c66ebd6d66e12990fc5db248b77c5b"
Sep 17 18:02:27.286476 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.286461    2583 scope.go:117] "RemoveContainer" containerID="d46807755338e662a2e8bfefdddceb52c8c9955d82ccafdd88fa6c0b6a27c6d7"
Sep 17 18:02:27.286743 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:02:27.286725    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d46807755338e662a2e8bfefdddceb52c8c9955d82ccafdd88fa6c0b6a27c6d7\": container with ID starting with d46807755338e662a2e8bfefdddceb52c8c9955d82ccafdd88fa6c0b6a27c6d7 not found: ID does not exist" containerID="d46807755338e662a2e8bfefdddceb52c8c9955d82ccafdd88fa6c0b6a27c6d7"
Sep 17 18:02:27.286796 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.286749    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d46807755338e662a2e8bfefdddceb52c8c9955d82ccafdd88fa6c0b6a27c6d7"} err="failed to get container status \"d46807755338e662a2e8bfefdddceb52c8c9955d82ccafdd88fa6c0b6a27c6d7\": rpc error: code = NotFound desc = could not find container \"d46807755338e662a2e8bfefdddceb52c8c9955d82ccafdd88fa6c0b6a27c6d7\": container with ID starting with d46807755338e662a2e8bfefdddceb52c8c9955d82ccafdd88fa6c0b6a27c6d7 not found: ID does not exist"
Sep 17 18:02:27.286796 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.286764    2583 scope.go:117] "RemoveContainer" containerID="fcfd10dc2334767448ed6304762c44a3f6c66ebd6d66e12990fc5db248b77c5b"
Sep 17 18:02:27.287022 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:02:27.287006    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcfd10dc2334767448ed6304762c44a3f6c66ebd6d66e12990fc5db248b77c5b\": container with ID starting with fcfd10dc2334767448ed6304762c44a3f6c66ebd6d66e12990fc5db248b77c5b not found: ID does not exist" containerID="fcfd10dc2334767448ed6304762c44a3f6c66ebd6d66e12990fc5db248b77c5b"
Sep 17 18:02:27.287068 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.287024    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcfd10dc2334767448ed6304762c44a3f6c66ebd6d66e12990fc5db248b77c5b"} err="failed to get container status \"fcfd10dc2334767448ed6304762c44a3f6c66ebd6d66e12990fc5db248b77c5b\": rpc error: code = NotFound desc = could not find container \"fcfd10dc2334767448ed6304762c44a3f6c66ebd6d66e12990fc5db248b77c5b\": container with ID starting with fcfd10dc2334767448ed6304762c44a3f6c66ebd6d66e12990fc5db248b77c5b not found: ID does not exist"
Sep 17 18:02:27.287068 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.287036    2583 scope.go:117] "RemoveContainer" containerID="d46807755338e662a2e8bfefdddceb52c8c9955d82ccafdd88fa6c0b6a27c6d7"
Sep 17 18:02:27.287242 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.287221    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d46807755338e662a2e8bfefdddceb52c8c9955d82ccafdd88fa6c0b6a27c6d7"} err="failed to get container status \"d46807755338e662a2e8bfefdddceb52c8c9955d82ccafdd88fa6c0b6a27c6d7\": rpc error: code = NotFound desc = could not find container \"d46807755338e662a2e8bfefdddceb52c8c9955d82ccafdd88fa6c0b6a27c6d7\": container with ID starting with d46807755338e662a2e8bfefdddceb52c8c9955d82ccafdd88fa6c0b6a27c6d7 not found: ID does not exist"
Sep 17 18:02:27.287282 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.287244    2583 scope.go:117] "RemoveContainer" containerID="fcfd10dc2334767448ed6304762c44a3f6c66ebd6d66e12990fc5db248b77c5b"
Sep 17 18:02:27.287449 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.287431    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcfd10dc2334767448ed6304762c44a3f6c66ebd6d66e12990fc5db248b77c5b"} err="failed to get container status \"fcfd10dc2334767448ed6304762c44a3f6c66ebd6d66e12990fc5db248b77c5b\": rpc error: code = NotFound desc = could not find container \"fcfd10dc2334767448ed6304762c44a3f6c66ebd6d66e12990fc5db248b77c5b\": container with ID starting with fcfd10dc2334767448ed6304762c44a3f6c66ebd6d66e12990fc5db248b77c5b not found: ID does not exist"
Sep 17 18:02:27.294348 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.294321    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"]
Sep 17 18:02:27.300064 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.300045    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-pldbw"]
Sep 17 18:02:27.325020 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.324998    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac5fbe45-640c-4550-b751-05153ab3a86f-dshm\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:02:27.325020 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.325018    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac5fbe45-640c-4550-b751-05153ab3a86f-model-cache\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:02:27.325110 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.325027    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac5fbe45-640c-4550-b751-05153ab3a86f-home\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:02:27.325110 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.325035    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac5fbe45-640c-4550-b751-05153ab3a86f-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:02:27.325110 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.325042    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac5fbe45-640c-4550-b751-05153ab3a86f-tmp-dir\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:02:27.325110 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:27.325051    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8vd5\" (UniqueName: \"kubernetes.io/projected/ac5fbe45-640c-4550-b751-05153ab3a86f-kube-api-access-q8vd5\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:02:29.039792 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:29.039757    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac5fbe45-640c-4550-b751-05153ab3a86f" path="/var/lib/kubelet/pods/ac5fbe45-640c-4550-b751-05153ab3a86f/volumes"
Sep 17 18:02:32.815052 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:32.815011    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl" podUID="81fcef74-838d-4177-9c15-aa421058f9cc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:02:41.689561 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:41.689512    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl" podUID="81fcef74-838d-4177-9c15-aa421058f9cc" containerName="main" containerID="cri-o://5df20b913f9491b25c7cb3f590fbcbd73112dbc65870cad6a4a7ccc802af363e" gracePeriod=45
Sep 17 18:02:42.814726 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:42.814678    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl" podUID="81fcef74-838d-4177-9c15-aa421058f9cc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:02:52.814369 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:52.814326    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl" podUID="81fcef74-838d-4177-9c15-aa421058f9cc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:02:52.814783 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:02:52.814441    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl"
Sep 17 18:03:02.814754 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:02.814652    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl" podUID="81fcef74-838d-4177-9c15-aa421058f9cc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:03:11.930496 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:11.930471    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl"
Sep 17 18:03:12.072241 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:12.072169    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81fcef74-838d-4177-9c15-aa421058f9cc-tls-certs\") pod \"81fcef74-838d-4177-9c15-aa421058f9cc\" (UID: \"81fcef74-838d-4177-9c15-aa421058f9cc\") "
Sep 17 18:03:12.072241 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:12.072222    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8w98\" (UniqueName: \"kubernetes.io/projected/81fcef74-838d-4177-9c15-aa421058f9cc-kube-api-access-s8w98\") pod \"81fcef74-838d-4177-9c15-aa421058f9cc\" (UID: \"81fcef74-838d-4177-9c15-aa421058f9cc\") "
Sep 17 18:03:12.074219 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:12.074188    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81fcef74-838d-4177-9c15-aa421058f9cc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "81fcef74-838d-4177-9c15-aa421058f9cc" (UID: "81fcef74-838d-4177-9c15-aa421058f9cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:03:12.074326 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:12.074247    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81fcef74-838d-4177-9c15-aa421058f9cc-kube-api-access-s8w98" (OuterVolumeSpecName: "kube-api-access-s8w98") pod "81fcef74-838d-4177-9c15-aa421058f9cc" (UID: "81fcef74-838d-4177-9c15-aa421058f9cc"). InnerVolumeSpecName "kube-api-access-s8w98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:03:12.173491 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:12.173464    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81fcef74-838d-4177-9c15-aa421058f9cc-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:03:12.173491 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:12.173488    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8w98\" (UniqueName: \"kubernetes.io/projected/81fcef74-838d-4177-9c15-aa421058f9cc-kube-api-access-s8w98\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:03:12.464312 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:12.464282    2583 generic.go:356] "Generic (PLEG): container finished" podID="81fcef74-838d-4177-9c15-aa421058f9cc" containerID="5df20b913f9491b25c7cb3f590fbcbd73112dbc65870cad6a4a7ccc802af363e" exitCode=0
Sep 17 18:03:12.464475 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:12.464331    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl"
Sep 17 18:03:12.464475 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:12.464369    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl" event={"ID":"81fcef74-838d-4177-9c15-aa421058f9cc","Type":"ContainerDied","Data":"5df20b913f9491b25c7cb3f590fbcbd73112dbc65870cad6a4a7ccc802af363e"}
Sep 17 18:03:12.464475 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:12.464426    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl" event={"ID":"81fcef74-838d-4177-9c15-aa421058f9cc","Type":"ContainerDied","Data":"552dc6ed979cb7b9e64d723e47c82cc07a68d4b337c9ed95f0fe4d152aec407e"}
Sep 17 18:03:12.464475 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:12.464457    2583 scope.go:117] "RemoveContainer" containerID="5df20b913f9491b25c7cb3f590fbcbd73112dbc65870cad6a4a7ccc802af363e"
Sep 17 18:03:12.473155 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:12.473141    2583 scope.go:117] "RemoveContainer" containerID="5df20b913f9491b25c7cb3f590fbcbd73112dbc65870cad6a4a7ccc802af363e"
Sep 17 18:03:12.473401 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:03:12.473383    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5df20b913f9491b25c7cb3f590fbcbd73112dbc65870cad6a4a7ccc802af363e\": container with ID starting with 5df20b913f9491b25c7cb3f590fbcbd73112dbc65870cad6a4a7ccc802af363e not found: ID does not exist" containerID="5df20b913f9491b25c7cb3f590fbcbd73112dbc65870cad6a4a7ccc802af363e"
Sep 17 18:03:12.473448 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:12.473407    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df20b913f9491b25c7cb3f590fbcbd73112dbc65870cad6a4a7ccc802af363e"} err="failed to get container status \"5df20b913f9491b25c7cb3f590fbcbd73112dbc65870cad6a4a7ccc802af363e\": rpc error: code = NotFound desc = could not find container \"5df20b913f9491b25c7cb3f590fbcbd73112dbc65870cad6a4a7ccc802af363e\": container with ID starting with 5df20b913f9491b25c7cb3f590fbcbd73112dbc65870cad6a4a7ccc802af363e not found: ID does not exist"
Sep 17 18:03:12.486823 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:12.486792    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl"]
Sep 17 18:03:12.491594 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:12.491563    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-757njrl"]
Sep 17 18:03:13.040042 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:13.040009    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81fcef74-838d-4177-9c15-aa421058f9cc" path="/var/lib/kubelet/pods/81fcef74-838d-4177-9c15-aa421058f9cc/volumes"
Sep 17 18:03:31.350795 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.350753    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"]
Sep 17 18:03:31.351276 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.351126    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81fcef74-838d-4177-9c15-aa421058f9cc" containerName="main"
Sep 17 18:03:31.351276 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.351140    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="81fcef74-838d-4177-9c15-aa421058f9cc" containerName="main"
Sep 17 18:03:31.351276 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.351156    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac5fbe45-640c-4550-b751-05153ab3a86f" containerName="main"
Sep 17 18:03:31.351276 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.351161    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac5fbe45-640c-4550-b751-05153ab3a86f" containerName="main"
Sep 17 18:03:31.351276 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.351171    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac5fbe45-640c-4550-b751-05153ab3a86f" containerName="llm-d-routing-sidecar"
Sep 17 18:03:31.351276 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.351176    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac5fbe45-640c-4550-b751-05153ab3a86f" containerName="llm-d-routing-sidecar"
Sep 17 18:03:31.351276 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.351239    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac5fbe45-640c-4550-b751-05153ab3a86f" containerName="llm-d-routing-sidecar"
Sep 17 18:03:31.351276 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.351251    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="81fcef74-838d-4177-9c15-aa421058f9cc" containerName="main"
Sep 17 18:03:31.351276 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.351261    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac5fbe45-640c-4550-b751-05153ab3a86f" containerName="main"
Sep 17 18:03:31.355924 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.355899    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"
Sep 17 18:03:31.358841 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.358810    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 17 18:03:31.358985 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.358886    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 17 18:03:31.359844 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.359824    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 17 18:03:31.359965 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.359824    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-tm579\""
Sep 17 18:03:31.359965 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.359885    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-zcqd8\""
Sep 17 18:03:31.374308 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.374279    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"]
Sep 17 18:03:31.422155 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.422122    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad94ad76-8111-4b78-b8dd-17ea58205cd8-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x\" (UID: \"ad94ad76-8111-4b78-b8dd-17ea58205cd8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"
Sep 17 18:03:31.422155 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.422156    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad94ad76-8111-4b78-b8dd-17ea58205cd8-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x\" (UID: \"ad94ad76-8111-4b78-b8dd-17ea58205cd8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"
Sep 17 18:03:31.422363 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.422190    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb7kf\" (UniqueName: \"kubernetes.io/projected/ad94ad76-8111-4b78-b8dd-17ea58205cd8-kube-api-access-hb7kf\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x\" (UID: \"ad94ad76-8111-4b78-b8dd-17ea58205cd8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"
Sep 17 18:03:31.422363 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.422274    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad94ad76-8111-4b78-b8dd-17ea58205cd8-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x\" (UID: \"ad94ad76-8111-4b78-b8dd-17ea58205cd8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"
Sep 17 18:03:31.422363 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.422320    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad94ad76-8111-4b78-b8dd-17ea58205cd8-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x\" (UID: \"ad94ad76-8111-4b78-b8dd-17ea58205cd8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"
Sep 17 18:03:31.422483 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.422361    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad94ad76-8111-4b78-b8dd-17ea58205cd8-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x\" (UID: \"ad94ad76-8111-4b78-b8dd-17ea58205cd8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"
Sep 17 18:03:31.523060 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.523028    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad94ad76-8111-4b78-b8dd-17ea58205cd8-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x\" (UID: \"ad94ad76-8111-4b78-b8dd-17ea58205cd8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"
Sep 17 18:03:31.523209 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.523064    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad94ad76-8111-4b78-b8dd-17ea58205cd8-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x\" (UID: \"ad94ad76-8111-4b78-b8dd-17ea58205cd8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"
Sep 17 18:03:31.523209 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.523126    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad94ad76-8111-4b78-b8dd-17ea58205cd8-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x\" (UID: \"ad94ad76-8111-4b78-b8dd-17ea58205cd8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"
Sep 17 18:03:31.523209 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.523151    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad94ad76-8111-4b78-b8dd-17ea58205cd8-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x\" (UID: \"ad94ad76-8111-4b78-b8dd-17ea58205cd8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"
Sep 17 18:03:31.523209 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.523186    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb7kf\" (UniqueName: \"kubernetes.io/projected/ad94ad76-8111-4b78-b8dd-17ea58205cd8-kube-api-access-hb7kf\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x\" (UID: \"ad94ad76-8111-4b78-b8dd-17ea58205cd8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"
Sep 17 18:03:31.523395 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.523362    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad94ad76-8111-4b78-b8dd-17ea58205cd8-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x\" (UID: \"ad94ad76-8111-4b78-b8dd-17ea58205cd8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"
Sep 17 18:03:31.523606 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.523568    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad94ad76-8111-4b78-b8dd-17ea58205cd8-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x\" (UID: \"ad94ad76-8111-4b78-b8dd-17ea58205cd8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"
Sep 17 18:03:31.523606 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.523584    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad94ad76-8111-4b78-b8dd-17ea58205cd8-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x\" (UID: \"ad94ad76-8111-4b78-b8dd-17ea58205cd8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"
Sep 17 18:03:31.523814 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.523680    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad94ad76-8111-4b78-b8dd-17ea58205cd8-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x\" (UID: \"ad94ad76-8111-4b78-b8dd-17ea58205cd8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"
Sep 17 18:03:31.525461 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.525438    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad94ad76-8111-4b78-b8dd-17ea58205cd8-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x\" (UID: \"ad94ad76-8111-4b78-b8dd-17ea58205cd8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"
Sep 17 18:03:31.525573 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.525556    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad94ad76-8111-4b78-b8dd-17ea58205cd8-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x\" (UID: \"ad94ad76-8111-4b78-b8dd-17ea58205cd8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"
Sep 17 18:03:31.532299 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.532279    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb7kf\" (UniqueName: \"kubernetes.io/projected/ad94ad76-8111-4b78-b8dd-17ea58205cd8-kube-api-access-hb7kf\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x\" (UID: \"ad94ad76-8111-4b78-b8dd-17ea58205cd8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"
Sep 17 18:03:31.667110 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.667025    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"
Sep 17 18:03:31.689010 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.688976    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4"]
Sep 17 18:03:31.692400 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.692381    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4"
Sep 17 18:03:31.694871 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.694849    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-pjh5m\""
Sep 17 18:03:31.702098 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.702078    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4"]
Sep 17 18:03:31.724877 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.724816    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrpbm\" (UniqueName: \"kubernetes.io/projected/ef403868-e049-4447-891e-65e438bcfc31-kube-api-access-vrpbm\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4\" (UID: \"ef403868-e049-4447-891e-65e438bcfc31\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4"
Sep 17 18:03:31.724999 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.724886    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef403868-e049-4447-891e-65e438bcfc31-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4\" (UID: \"ef403868-e049-4447-891e-65e438bcfc31\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4"
Sep 17 18:03:31.825854 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.825817    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrpbm\" (UniqueName: \"kubernetes.io/projected/ef403868-e049-4447-891e-65e438bcfc31-kube-api-access-vrpbm\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4\" (UID: \"ef403868-e049-4447-891e-65e438bcfc31\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4"
Sep 17 18:03:31.826006 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.825869    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef403868-e049-4447-891e-65e438bcfc31-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4\" (UID: \"ef403868-e049-4447-891e-65e438bcfc31\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4"
Sep 17 18:03:31.828252 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.828233    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef403868-e049-4447-891e-65e438bcfc31-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4\" (UID: \"ef403868-e049-4447-891e-65e438bcfc31\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4"
Sep 17 18:03:31.835477 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.835456    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrpbm\" (UniqueName: \"kubernetes.io/projected/ef403868-e049-4447-891e-65e438bcfc31-kube-api-access-vrpbm\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4\" (UID: \"ef403868-e049-4447-891e-65e438bcfc31\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4"
Sep 17 18:03:31.950154 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:31.950070    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"]
Sep 17 18:03:31.952846 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:03:31.952819    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad94ad76_8111_4b78_b8dd_17ea58205cd8.slice/crio-c778bbf54857b765a3f076bf799daec05634a39ce5d9756410dc47c79506d419 WatchSource:0}: Error finding container c778bbf54857b765a3f076bf799daec05634a39ce5d9756410dc47c79506d419: Status 404 returned error can't find the container with id c778bbf54857b765a3f076bf799daec05634a39ce5d9756410dc47c79506d419
Sep 17 18:03:32.003147 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:32.003116    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4"
Sep 17 18:03:32.275312 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:32.275173    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4"]
Sep 17 18:03:32.279822 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:03:32.279787    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef403868_e049_4447_891e_65e438bcfc31.slice/crio-cae442aec10873a3501e94807e4368595a57269cfe0966bf6ab9211fa5a1e340 WatchSource:0}: Error finding container cae442aec10873a3501e94807e4368595a57269cfe0966bf6ab9211fa5a1e340: Status 404 returned error can't find the container with id cae442aec10873a3501e94807e4368595a57269cfe0966bf6ab9211fa5a1e340
Sep 17 18:03:32.546765 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:32.546686    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x" event={"ID":"ad94ad76-8111-4b78-b8dd-17ea58205cd8","Type":"ContainerStarted","Data":"340d1ca7d512ed91804cb7381691606d66e5321ac889d7d6b644b55b2dd61c08"}
Sep 17 18:03:32.546765 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:32.546749    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x" event={"ID":"ad94ad76-8111-4b78-b8dd-17ea58205cd8","Type":"ContainerStarted","Data":"c778bbf54857b765a3f076bf799daec05634a39ce5d9756410dc47c79506d419"}
Sep 17 18:03:32.547188 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:32.546807    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"
Sep 17 18:03:32.548606 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:32.548541    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4" event={"ID":"ef403868-e049-4447-891e-65e438bcfc31","Type":"ContainerStarted","Data":"3cb54ff57afae732ca0db194cc1f7ebe47e3f5c71d2e45d5d1a42418d4203f44"}
Sep 17 18:03:32.548606 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:32.548580    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4" event={"ID":"ef403868-e049-4447-891e-65e438bcfc31","Type":"ContainerStarted","Data":"cae442aec10873a3501e94807e4368595a57269cfe0966bf6ab9211fa5a1e340"}
Sep 17 18:03:32.548771 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:32.548615    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4"
Sep 17 18:03:32.588611 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:32.588567    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4" podStartSLOduration=1.588553484 podStartE2EDuration="1.588553484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:32.588204024 +0000 UTC m=+1723.679510804" watchObservedRunningTime="2026-09-17 18:03:32.588553484 +0000 UTC m=+1723.679860245"
Sep 17 18:03:33.554781 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:33.554748    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x" event={"ID":"ad94ad76-8111-4b78-b8dd-17ea58205cd8","Type":"ContainerStarted","Data":"4f07727d9fcea602fb52e53cfe899bc471ae4966cb215f820c0e47d2b7dbee57"}
Sep 17 18:03:33.577971 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:33.577926    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x" podStartSLOduration=2.5779125179999998 podStartE2EDuration="2.577912518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:33.576321097 +0000 UTC m=+1724.667627887" watchObservedRunningTime="2026-09-17 18:03:33.577912518 +0000 UTC m=+1724.669219283"
Sep 17 18:03:33.667854 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:33.667830    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"
Sep 17 18:03:41.667531 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:41.667496    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"
Sep 17 18:03:41.679779 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:41.679758    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"
Sep 17 18:03:42.604067 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:42.604037    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"
Sep 17 18:03:42.605257 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:03:42.605238    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"
Sep 17 18:04:03.556473 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:03.556443    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4"
Sep 17 18:04:07.188532 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.188497    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4"]
Sep 17 18:04:07.192374 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.192344    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"]
Sep 17 18:04:07.200484 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:04:07.200449    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:04:07.200484 ip-10-0-141-30 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:04:07.200484 ip-10-0-141-30 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 18:04:07.200484 ip-10-0-141-30 kubenswrapper[2583]:  > containerID="4f07727d9fcea602fb52e53cfe899bc471ae4966cb215f820c0e47d2b7dbee57" cmd=["/bin/sleep","15"]
Sep 17 18:04:07.200732 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.200507    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x" message="" err=<
Sep 17 18:04:07.200732 ip-10-0-141-30 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:04:07.200732 ip-10-0-141-30 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 18:04:07.200732 ip-10-0-141-30 kubenswrapper[2583]:  >
Sep 17 18:04:07.200732 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:04:07.200523    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:04:07.200732 ip-10-0-141-30 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:04:07.200732 ip-10-0-141-30 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 18:04:07.200732 ip-10-0-141-30 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x" podUID="ad94ad76-8111-4b78-b8dd-17ea58205cd8" containerName="main" containerID="cri-o://4f07727d9fcea602fb52e53cfe899bc471ae4966cb215f820c0e47d2b7dbee57"
Sep 17 18:04:07.200732 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.200563    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x" podUID="ad94ad76-8111-4b78-b8dd-17ea58205cd8" containerName="main" containerID="cri-o://4f07727d9fcea602fb52e53cfe899bc471ae4966cb215f820c0e47d2b7dbee57" gracePeriod=60
Sep 17 18:04:07.415980 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.415948    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x" podUID="ad94ad76-8111-4b78-b8dd-17ea58205cd8" containerName="llm-d-routing-sidecar" containerID="cri-o://340d1ca7d512ed91804cb7381691606d66e5321ac889d7d6b644b55b2dd61c08" gracePeriod=60
Sep 17 18:04:07.655740 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.655718    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"
Sep 17 18:04:07.707428 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.707393    2583 generic.go:356] "Generic (PLEG): container finished" podID="ad94ad76-8111-4b78-b8dd-17ea58205cd8" containerID="4f07727d9fcea602fb52e53cfe899bc471ae4966cb215f820c0e47d2b7dbee57" exitCode=0
Sep 17 18:04:07.707428 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.707421    2583 generic.go:356] "Generic (PLEG): container finished" podID="ad94ad76-8111-4b78-b8dd-17ea58205cd8" containerID="340d1ca7d512ed91804cb7381691606d66e5321ac889d7d6b644b55b2dd61c08" exitCode=0
Sep 17 18:04:07.707617 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.707427    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x" event={"ID":"ad94ad76-8111-4b78-b8dd-17ea58205cd8","Type":"ContainerDied","Data":"4f07727d9fcea602fb52e53cfe899bc471ae4966cb215f820c0e47d2b7dbee57"}
Sep 17 18:04:07.707617 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.707450    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"
Sep 17 18:04:07.707617 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.707474    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x" event={"ID":"ad94ad76-8111-4b78-b8dd-17ea58205cd8","Type":"ContainerDied","Data":"340d1ca7d512ed91804cb7381691606d66e5321ac889d7d6b644b55b2dd61c08"}
Sep 17 18:04:07.707617 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.707495    2583 scope.go:117] "RemoveContainer" containerID="4f07727d9fcea602fb52e53cfe899bc471ae4966cb215f820c0e47d2b7dbee57"
Sep 17 18:04:07.707617 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.707554    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x" event={"ID":"ad94ad76-8111-4b78-b8dd-17ea58205cd8","Type":"ContainerDied","Data":"c778bbf54857b765a3f076bf799daec05634a39ce5d9756410dc47c79506d419"}
Sep 17 18:04:07.715690 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.715654    2583 scope.go:117] "RemoveContainer" containerID="340d1ca7d512ed91804cb7381691606d66e5321ac889d7d6b644b55b2dd61c08"
Sep 17 18:04:07.722720 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.722702    2583 scope.go:117] "RemoveContainer" containerID="4f07727d9fcea602fb52e53cfe899bc471ae4966cb215f820c0e47d2b7dbee57"
Sep 17 18:04:07.722967 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:04:07.722948    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f07727d9fcea602fb52e53cfe899bc471ae4966cb215f820c0e47d2b7dbee57\": container with ID starting with 4f07727d9fcea602fb52e53cfe899bc471ae4966cb215f820c0e47d2b7dbee57 not found: ID does not exist" containerID="4f07727d9fcea602fb52e53cfe899bc471ae4966cb215f820c0e47d2b7dbee57"
Sep 17 18:04:07.723031 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.722975    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f07727d9fcea602fb52e53cfe899bc471ae4966cb215f820c0e47d2b7dbee57"} err="failed to get container status \"4f07727d9fcea602fb52e53cfe899bc471ae4966cb215f820c0e47d2b7dbee57\": rpc error: code = NotFound desc = could not find container \"4f07727d9fcea602fb52e53cfe899bc471ae4966cb215f820c0e47d2b7dbee57\": container with ID starting with 4f07727d9fcea602fb52e53cfe899bc471ae4966cb215f820c0e47d2b7dbee57 not found: ID does not exist"
Sep 17 18:04:07.723031 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.722995    2583 scope.go:117] "RemoveContainer" containerID="340d1ca7d512ed91804cb7381691606d66e5321ac889d7d6b644b55b2dd61c08"
Sep 17 18:04:07.723247 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:04:07.723230    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"340d1ca7d512ed91804cb7381691606d66e5321ac889d7d6b644b55b2dd61c08\": container with ID starting with 340d1ca7d512ed91804cb7381691606d66e5321ac889d7d6b644b55b2dd61c08 not found: ID does not exist" containerID="340d1ca7d512ed91804cb7381691606d66e5321ac889d7d6b644b55b2dd61c08"
Sep 17 18:04:07.723283 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.723251    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"340d1ca7d512ed91804cb7381691606d66e5321ac889d7d6b644b55b2dd61c08"} err="failed to get container status \"340d1ca7d512ed91804cb7381691606d66e5321ac889d7d6b644b55b2dd61c08\": rpc error: code = NotFound desc = could not find container \"340d1ca7d512ed91804cb7381691606d66e5321ac889d7d6b644b55b2dd61c08\": container with ID starting with 340d1ca7d512ed91804cb7381691606d66e5321ac889d7d6b644b55b2dd61c08 not found: ID does not exist"
Sep 17 18:04:07.723283 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.723264    2583 scope.go:117] "RemoveContainer" containerID="4f07727d9fcea602fb52e53cfe899bc471ae4966cb215f820c0e47d2b7dbee57"
Sep 17 18:04:07.723514 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.723495    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f07727d9fcea602fb52e53cfe899bc471ae4966cb215f820c0e47d2b7dbee57"} err="failed to get container status \"4f07727d9fcea602fb52e53cfe899bc471ae4966cb215f820c0e47d2b7dbee57\": rpc error: code = NotFound desc = could not find container \"4f07727d9fcea602fb52e53cfe899bc471ae4966cb215f820c0e47d2b7dbee57\": container with ID starting with 4f07727d9fcea602fb52e53cfe899bc471ae4966cb215f820c0e47d2b7dbee57 not found: ID does not exist"
Sep 17 18:04:07.723571 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.723515    2583 scope.go:117] "RemoveContainer" containerID="340d1ca7d512ed91804cb7381691606d66e5321ac889d7d6b644b55b2dd61c08"
Sep 17 18:04:07.723772 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.723756    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"340d1ca7d512ed91804cb7381691606d66e5321ac889d7d6b644b55b2dd61c08"} err="failed to get container status \"340d1ca7d512ed91804cb7381691606d66e5321ac889d7d6b644b55b2dd61c08\": rpc error: code = NotFound desc = could not find container \"340d1ca7d512ed91804cb7381691606d66e5321ac889d7d6b644b55b2dd61c08\": container with ID starting with 340d1ca7d512ed91804cb7381691606d66e5321ac889d7d6b644b55b2dd61c08 not found: ID does not exist"
Sep 17 18:04:07.807778 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.807706    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad94ad76-8111-4b78-b8dd-17ea58205cd8-model-cache\") pod \"ad94ad76-8111-4b78-b8dd-17ea58205cd8\" (UID: \"ad94ad76-8111-4b78-b8dd-17ea58205cd8\") "
Sep 17 18:04:07.807778 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.807756    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad94ad76-8111-4b78-b8dd-17ea58205cd8-tmp-dir\") pod \"ad94ad76-8111-4b78-b8dd-17ea58205cd8\" (UID: \"ad94ad76-8111-4b78-b8dd-17ea58205cd8\") "
Sep 17 18:04:07.807952 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.807796    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hb7kf\" (UniqueName: \"kubernetes.io/projected/ad94ad76-8111-4b78-b8dd-17ea58205cd8-kube-api-access-hb7kf\") pod \"ad94ad76-8111-4b78-b8dd-17ea58205cd8\" (UID: \"ad94ad76-8111-4b78-b8dd-17ea58205cd8\") "
Sep 17 18:04:07.807952 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.807858    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad94ad76-8111-4b78-b8dd-17ea58205cd8-home\") pod \"ad94ad76-8111-4b78-b8dd-17ea58205cd8\" (UID: \"ad94ad76-8111-4b78-b8dd-17ea58205cd8\") "
Sep 17 18:04:07.807952 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.807896    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad94ad76-8111-4b78-b8dd-17ea58205cd8-tls-certs\") pod \"ad94ad76-8111-4b78-b8dd-17ea58205cd8\" (UID: \"ad94ad76-8111-4b78-b8dd-17ea58205cd8\") "
Sep 17 18:04:07.807952 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.807920    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad94ad76-8111-4b78-b8dd-17ea58205cd8-dshm\") pod \"ad94ad76-8111-4b78-b8dd-17ea58205cd8\" (UID: \"ad94ad76-8111-4b78-b8dd-17ea58205cd8\") "
Sep 17 18:04:07.808129 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.808015    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad94ad76-8111-4b78-b8dd-17ea58205cd8-model-cache" (OuterVolumeSpecName: "model-cache") pod "ad94ad76-8111-4b78-b8dd-17ea58205cd8" (UID: "ad94ad76-8111-4b78-b8dd-17ea58205cd8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:07.808129 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.808109    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad94ad76-8111-4b78-b8dd-17ea58205cd8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ad94ad76-8111-4b78-b8dd-17ea58205cd8" (UID: "ad94ad76-8111-4b78-b8dd-17ea58205cd8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:07.808212 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.808151    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad94ad76-8111-4b78-b8dd-17ea58205cd8-home" (OuterVolumeSpecName: "home") pod "ad94ad76-8111-4b78-b8dd-17ea58205cd8" (UID: "ad94ad76-8111-4b78-b8dd-17ea58205cd8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:07.808212 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.808162    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad94ad76-8111-4b78-b8dd-17ea58205cd8-model-cache\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:04:07.810067 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.810042    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad94ad76-8111-4b78-b8dd-17ea58205cd8-dshm" (OuterVolumeSpecName: "dshm") pod "ad94ad76-8111-4b78-b8dd-17ea58205cd8" (UID: "ad94ad76-8111-4b78-b8dd-17ea58205cd8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:07.810170 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.810079    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad94ad76-8111-4b78-b8dd-17ea58205cd8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ad94ad76-8111-4b78-b8dd-17ea58205cd8" (UID: "ad94ad76-8111-4b78-b8dd-17ea58205cd8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:04:07.810170 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.810151    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad94ad76-8111-4b78-b8dd-17ea58205cd8-kube-api-access-hb7kf" (OuterVolumeSpecName: "kube-api-access-hb7kf") pod "ad94ad76-8111-4b78-b8dd-17ea58205cd8" (UID: "ad94ad76-8111-4b78-b8dd-17ea58205cd8"). InnerVolumeSpecName "kube-api-access-hb7kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:04:07.909000 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.908956    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb7kf\" (UniqueName: \"kubernetes.io/projected/ad94ad76-8111-4b78-b8dd-17ea58205cd8-kube-api-access-hb7kf\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:04:07.909000 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.908995    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad94ad76-8111-4b78-b8dd-17ea58205cd8-home\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:04:07.909000 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.909005    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad94ad76-8111-4b78-b8dd-17ea58205cd8-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:04:07.909000 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.909014    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad94ad76-8111-4b78-b8dd-17ea58205cd8-dshm\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:04:07.909258 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:07.909022    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad94ad76-8111-4b78-b8dd-17ea58205cd8-tmp-dir\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:04:08.028305 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:08.028273    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"]
Sep 17 18:04:08.036244 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:08.036212    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x"]
Sep 17 18:04:08.595013 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:08.594967    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-n7m5x" podUID="ad94ad76-8111-4b78-b8dd-17ea58205cd8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8001/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 18:04:09.040236 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:09.040163    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad94ad76-8111-4b78-b8dd-17ea58205cd8" path="/var/lib/kubelet/pods/ad94ad76-8111-4b78-b8dd-17ea58205cd8/volumes"
Sep 17 18:04:13.557078 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:13.557038    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4" podUID="ef403868-e049-4447-891e-65e438bcfc31" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:04:22.207950 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:22.207908    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4" podUID="ef403868-e049-4447-891e-65e438bcfc31" containerName="main" containerID="cri-o://3cb54ff57afae732ca0db194cc1f7ebe47e3f5c71d2e45d5d1a42418d4203f44" gracePeriod=45
Sep 17 18:04:23.556710 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:23.556671    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4" podUID="ef403868-e049-4447-891e-65e438bcfc31" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:04:33.556923 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:33.556889    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4" podUID="ef403868-e049-4447-891e-65e438bcfc31" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:04:33.557321 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:33.556992    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4"
Sep 17 18:04:43.556852 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:43.556817    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4" podUID="ef403868-e049-4447-891e-65e438bcfc31" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:04:52.455372 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:52.455342    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4"
Sep 17 18:04:52.536005 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:52.535944    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrpbm\" (UniqueName: \"kubernetes.io/projected/ef403868-e049-4447-891e-65e438bcfc31-kube-api-access-vrpbm\") pod \"ef403868-e049-4447-891e-65e438bcfc31\" (UID: \"ef403868-e049-4447-891e-65e438bcfc31\") "
Sep 17 18:04:52.536005 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:52.535984    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef403868-e049-4447-891e-65e438bcfc31-tls-certs\") pod \"ef403868-e049-4447-891e-65e438bcfc31\" (UID: \"ef403868-e049-4447-891e-65e438bcfc31\") "
Sep 17 18:04:52.537946 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:52.537913    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef403868-e049-4447-891e-65e438bcfc31-kube-api-access-vrpbm" (OuterVolumeSpecName: "kube-api-access-vrpbm") pod "ef403868-e049-4447-891e-65e438bcfc31" (UID: "ef403868-e049-4447-891e-65e438bcfc31"). InnerVolumeSpecName "kube-api-access-vrpbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:04:52.538057 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:52.538000    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef403868-e049-4447-891e-65e438bcfc31-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ef403868-e049-4447-891e-65e438bcfc31" (UID: "ef403868-e049-4447-891e-65e438bcfc31"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:04:52.636593 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:52.636562    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrpbm\" (UniqueName: \"kubernetes.io/projected/ef403868-e049-4447-891e-65e438bcfc31-kube-api-access-vrpbm\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:04:52.636593 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:52.636590    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef403868-e049-4447-891e-65e438bcfc31-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:04:52.888554 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:52.888517    2583 generic.go:356] "Generic (PLEG): container finished" podID="ef403868-e049-4447-891e-65e438bcfc31" containerID="3cb54ff57afae732ca0db194cc1f7ebe47e3f5c71d2e45d5d1a42418d4203f44" exitCode=0
Sep 17 18:04:52.888740 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:52.888562    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4"
Sep 17 18:04:52.888740 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:52.888558    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4" event={"ID":"ef403868-e049-4447-891e-65e438bcfc31","Type":"ContainerDied","Data":"3cb54ff57afae732ca0db194cc1f7ebe47e3f5c71d2e45d5d1a42418d4203f44"}
Sep 17 18:04:52.888740 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:52.888702    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4" event={"ID":"ef403868-e049-4447-891e-65e438bcfc31","Type":"ContainerDied","Data":"cae442aec10873a3501e94807e4368595a57269cfe0966bf6ab9211fa5a1e340"}
Sep 17 18:04:52.888740 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:52.888726    2583 scope.go:117] "RemoveContainer" containerID="3cb54ff57afae732ca0db194cc1f7ebe47e3f5c71d2e45d5d1a42418d4203f44"
Sep 17 18:04:52.897070 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:52.897054    2583 scope.go:117] "RemoveContainer" containerID="3cb54ff57afae732ca0db194cc1f7ebe47e3f5c71d2e45d5d1a42418d4203f44"
Sep 17 18:04:52.897338 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:04:52.897321    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cb54ff57afae732ca0db194cc1f7ebe47e3f5c71d2e45d5d1a42418d4203f44\": container with ID starting with 3cb54ff57afae732ca0db194cc1f7ebe47e3f5c71d2e45d5d1a42418d4203f44 not found: ID does not exist" containerID="3cb54ff57afae732ca0db194cc1f7ebe47e3f5c71d2e45d5d1a42418d4203f44"
Sep 17 18:04:52.897384 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:52.897347    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cb54ff57afae732ca0db194cc1f7ebe47e3f5c71d2e45d5d1a42418d4203f44"} err="failed to get container status \"3cb54ff57afae732ca0db194cc1f7ebe47e3f5c71d2e45d5d1a42418d4203f44\": rpc error: code = NotFound desc = could not find container \"3cb54ff57afae732ca0db194cc1f7ebe47e3f5c71d2e45d5d1a42418d4203f44\": container with ID starting with 3cb54ff57afae732ca0db194cc1f7ebe47e3f5c71d2e45d5d1a42418d4203f44 not found: ID does not exist"
Sep 17 18:04:52.909038 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:52.909017    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4"]
Sep 17 18:04:52.913385 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:52.913364    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedqv5c4"]
Sep 17 18:04:53.039617 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:04:53.039590    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef403868-e049-4447-891e-65e438bcfc31" path="/var/lib/kubelet/pods/ef403868-e049-4447-891e-65e438bcfc31/volumes"
Sep 17 18:05:07.189015 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.188979    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"]
Sep 17 18:05:07.189419 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.189278    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad94ad76-8111-4b78-b8dd-17ea58205cd8" containerName="main"
Sep 17 18:05:07.189419 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.189290    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad94ad76-8111-4b78-b8dd-17ea58205cd8" containerName="main"
Sep 17 18:05:07.189419 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.189317    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad94ad76-8111-4b78-b8dd-17ea58205cd8" containerName="llm-d-routing-sidecar"
Sep 17 18:05:07.189419 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.189328    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad94ad76-8111-4b78-b8dd-17ea58205cd8" containerName="llm-d-routing-sidecar"
Sep 17 18:05:07.189419 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.189336    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef403868-e049-4447-891e-65e438bcfc31" containerName="main"
Sep 17 18:05:07.189419 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.189341    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef403868-e049-4447-891e-65e438bcfc31" containerName="main"
Sep 17 18:05:07.189419 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.189392    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad94ad76-8111-4b78-b8dd-17ea58205cd8" containerName="llm-d-routing-sidecar"
Sep 17 18:05:07.189419 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.189403    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef403868-e049-4447-891e-65e438bcfc31" containerName="main"
Sep 17 18:05:07.189419 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.189408    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad94ad76-8111-4b78-b8dd-17ea58205cd8" containerName="main"
Sep 17 18:05:07.193684 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.193652    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"
Sep 17 18:05:07.197573 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.197552    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:05:07.197675 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.197588    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 17 18:05:07.198719 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.198627    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 17 18:05:07.198719 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.198650    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-9wzmx\""
Sep 17 18:05:07.198880 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.198721    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 17 18:05:07.198880 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.198719    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 17 18:05:07.210409 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.210389    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"]
Sep 17 18:05:07.249930 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.249901    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e610c8d-5c8b-474f-b597-1e0e81c4b224-model-cache\") pod \"tokenizer-clean-path-test-kserve-5b769465b9-l6nzr\" (UID: \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"
Sep 17 18:05:07.250060 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.249933    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e610c8d-5c8b-474f-b597-1e0e81c4b224-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-5b769465b9-l6nzr\" (UID: \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"
Sep 17 18:05:07.250060 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.249961    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e610c8d-5c8b-474f-b597-1e0e81c4b224-home\") pod \"tokenizer-clean-path-test-kserve-5b769465b9-l6nzr\" (UID: \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"
Sep 17 18:05:07.250168 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.250055    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e610c8d-5c8b-474f-b597-1e0e81c4b224-dshm\") pod \"tokenizer-clean-path-test-kserve-5b769465b9-l6nzr\" (UID: \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"
Sep 17 18:05:07.250168 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.250088    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e610c8d-5c8b-474f-b597-1e0e81c4b224-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5b769465b9-l6nzr\" (UID: \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"
Sep 17 18:05:07.250168 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.250119    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e610c8d-5c8b-474f-b597-1e0e81c4b224-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5b769465b9-l6nzr\" (UID: \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"
Sep 17 18:05:07.250168 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.250143    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd4tx\" (UniqueName: \"kubernetes.io/projected/5e610c8d-5c8b-474f-b597-1e0e81c4b224-kube-api-access-hd4tx\") pod \"tokenizer-clean-path-test-kserve-5b769465b9-l6nzr\" (UID: \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"
Sep 17 18:05:07.250333 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.250268    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e610c8d-5c8b-474f-b597-1e0e81c4b224-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5b769465b9-l6nzr\" (UID: \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"
Sep 17 18:05:07.351462 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.351430    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e610c8d-5c8b-474f-b597-1e0e81c4b224-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5b769465b9-l6nzr\" (UID: \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"
Sep 17 18:05:07.351587 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.351475    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e610c8d-5c8b-474f-b597-1e0e81c4b224-model-cache\") pod \"tokenizer-clean-path-test-kserve-5b769465b9-l6nzr\" (UID: \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"
Sep 17 18:05:07.351587 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.351501    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e610c8d-5c8b-474f-b597-1e0e81c4b224-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-5b769465b9-l6nzr\" (UID: \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"
Sep 17 18:05:07.351587 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.351534    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e610c8d-5c8b-474f-b597-1e0e81c4b224-home\") pod \"tokenizer-clean-path-test-kserve-5b769465b9-l6nzr\" (UID: \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"
Sep 17 18:05:07.351587 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.351561    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e610c8d-5c8b-474f-b597-1e0e81c4b224-dshm\") pod \"tokenizer-clean-path-test-kserve-5b769465b9-l6nzr\" (UID: \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"
Sep 17 18:05:07.351587 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.351585    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e610c8d-5c8b-474f-b597-1e0e81c4b224-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5b769465b9-l6nzr\" (UID: \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"
Sep 17 18:05:07.351870 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.351613    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e610c8d-5c8b-474f-b597-1e0e81c4b224-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5b769465b9-l6nzr\" (UID: \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"
Sep 17 18:05:07.351870 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.351637    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd4tx\" (UniqueName: \"kubernetes.io/projected/5e610c8d-5c8b-474f-b597-1e0e81c4b224-kube-api-access-hd4tx\") pod \"tokenizer-clean-path-test-kserve-5b769465b9-l6nzr\" (UID: \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"
Sep 17 18:05:07.351978 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.351939    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e610c8d-5c8b-474f-b597-1e0e81c4b224-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5b769465b9-l6nzr\" (UID: \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"
Sep 17 18:05:07.352046 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.351977    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e610c8d-5c8b-474f-b597-1e0e81c4b224-home\") pod \"tokenizer-clean-path-test-kserve-5b769465b9-l6nzr\" (UID: \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"
Sep 17 18:05:07.352093 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.352057    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e610c8d-5c8b-474f-b597-1e0e81c4b224-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5b769465b9-l6nzr\" (UID: \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"
Sep 17 18:05:07.352251 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.352231    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e610c8d-5c8b-474f-b597-1e0e81c4b224-model-cache\") pod \"tokenizer-clean-path-test-kserve-5b769465b9-l6nzr\" (UID: \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"
Sep 17 18:05:07.352345 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.352327    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e610c8d-5c8b-474f-b597-1e0e81c4b224-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-5b769465b9-l6nzr\" (UID: \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"
Sep 17 18:05:07.353859 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.353838    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e610c8d-5c8b-474f-b597-1e0e81c4b224-dshm\") pod \"tokenizer-clean-path-test-kserve-5b769465b9-l6nzr\" (UID: \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"
Sep 17 18:05:07.354185 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.354158    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e610c8d-5c8b-474f-b597-1e0e81c4b224-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5b769465b9-l6nzr\" (UID: \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"
Sep 17 18:05:07.359595 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.359569    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd4tx\" (UniqueName: \"kubernetes.io/projected/5e610c8d-5c8b-474f-b597-1e0e81c4b224-kube-api-access-hd4tx\") pod \"tokenizer-clean-path-test-kserve-5b769465b9-l6nzr\" (UID: \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"
Sep 17 18:05:07.503689 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.503575    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"
Sep 17 18:05:07.550694 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.550644    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll"]
Sep 17 18:05:07.556415 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.556391    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll"
Sep 17 18:05:07.559128 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.559108    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-whjn7\""
Sep 17 18:05:07.565144 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.565119    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll"]
Sep 17 18:05:07.654460 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.654428    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbcw9\" (UniqueName: \"kubernetes.io/projected/45b08415-30f1-4a74-bdf5-56292806f668-kube-api-access-qbcw9\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll\" (UID: \"45b08415-30f1-4a74-bdf5-56292806f668\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll"
Sep 17 18:05:07.654616 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.654500    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b08415-30f1-4a74-bdf5-56292806f668-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll\" (UID: \"45b08415-30f1-4a74-bdf5-56292806f668\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll"
Sep 17 18:05:07.755069 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.754988    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b08415-30f1-4a74-bdf5-56292806f668-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll\" (UID: \"45b08415-30f1-4a74-bdf5-56292806f668\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll"
Sep 17 18:05:07.755069 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.755042    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbcw9\" (UniqueName: \"kubernetes.io/projected/45b08415-30f1-4a74-bdf5-56292806f668-kube-api-access-qbcw9\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll\" (UID: \"45b08415-30f1-4a74-bdf5-56292806f668\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll"
Sep 17 18:05:07.757619 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.757591    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b08415-30f1-4a74-bdf5-56292806f668-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll\" (UID: \"45b08415-30f1-4a74-bdf5-56292806f668\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll"
Sep 17 18:05:07.762943 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.762917    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbcw9\" (UniqueName: \"kubernetes.io/projected/45b08415-30f1-4a74-bdf5-56292806f668-kube-api-access-qbcw9\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll\" (UID: \"45b08415-30f1-4a74-bdf5-56292806f668\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll"
Sep 17 18:05:07.789415 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.789396    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"]
Sep 17 18:05:07.791911 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:05:07.791885    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e610c8d_5c8b_474f_b597_1e0e81c4b224.slice/crio-3b588b253b8e17d60b9092f5e7dd2b18fd2f4a5769bcdbe67ec8836339f31662 WatchSource:0}: Error finding container 3b588b253b8e17d60b9092f5e7dd2b18fd2f4a5769bcdbe67ec8836339f31662: Status 404 returned error can't find the container with id 3b588b253b8e17d60b9092f5e7dd2b18fd2f4a5769bcdbe67ec8836339f31662
Sep 17 18:05:07.873078 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.873043    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll"
Sep 17 18:05:07.955307 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.955276    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr" event={"ID":"5e610c8d-5c8b-474f-b597-1e0e81c4b224","Type":"ContainerStarted","Data":"47203a33169e1ab93a59f2448fa2bb6b0f32acd74ccd252ec6e4696e24ba0b22"}
Sep 17 18:05:07.955452 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:07.955348    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr" event={"ID":"5e610c8d-5c8b-474f-b597-1e0e81c4b224","Type":"ContainerStarted","Data":"3b588b253b8e17d60b9092f5e7dd2b18fd2f4a5769bcdbe67ec8836339f31662"}
Sep 17 18:05:08.159300 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:08.159264    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll"]
Sep 17 18:05:08.162629 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:05:08.162600    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b08415_30f1_4a74_bdf5_56292806f668.slice/crio-8d90164f11029e0fb4c50a935fd080257d81ad622b1baac10225e4d54d6b6c74 WatchSource:0}: Error finding container 8d90164f11029e0fb4c50a935fd080257d81ad622b1baac10225e4d54d6b6c74: Status 404 returned error can't find the container with id 8d90164f11029e0fb4c50a935fd080257d81ad622b1baac10225e4d54d6b6c74
Sep 17 18:05:08.961329 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:08.961286    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll" event={"ID":"45b08415-30f1-4a74-bdf5-56292806f668","Type":"ContainerStarted","Data":"30a5f51c710a60b145add1e62be9bbb385274232ecbdbdafe73b67fd420f1c90"}
Sep 17 18:05:08.961820 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:08.961392    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll" event={"ID":"45b08415-30f1-4a74-bdf5-56292806f668","Type":"ContainerStarted","Data":"8d90164f11029e0fb4c50a935fd080257d81ad622b1baac10225e4d54d6b6c74"}
Sep 17 18:05:08.981307 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:08.981249    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll" podStartSLOduration=1.981227241 podStartE2EDuration="1.981227241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:05:08.979802082 +0000 UTC m=+1820.071108865" watchObservedRunningTime="2026-09-17 18:05:08.981227241 +0000 UTC m=+1820.072534007"
Sep 17 18:05:09.965556 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:09.965517    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll"
Sep 17 18:05:10.970923 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:10.970892    2583 generic.go:356] "Generic (PLEG): container finished" podID="5e610c8d-5c8b-474f-b597-1e0e81c4b224" containerID="47203a33169e1ab93a59f2448fa2bb6b0f32acd74ccd252ec6e4696e24ba0b22" exitCode=0
Sep 17 18:05:10.971339 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:10.970973    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr" event={"ID":"5e610c8d-5c8b-474f-b597-1e0e81c4b224","Type":"ContainerDied","Data":"47203a33169e1ab93a59f2448fa2bb6b0f32acd74ccd252ec6e4696e24ba0b22"}
Sep 17 18:05:11.977427 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:11.977392    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr" event={"ID":"5e610c8d-5c8b-474f-b597-1e0e81c4b224","Type":"ContainerStarted","Data":"bdd2e9aad6ceb87727c105c734ead2323c44dc6646bd809257bc85dd0cb2c33a"}
Sep 17 18:05:12.004477 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:12.004431    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr" podStartSLOduration=2.87419345 podStartE2EDuration="5.004415746s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.130222296s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:05:12.002382704 +0000 UTC m=+1823.093689482" watchObservedRunningTime="2026-09-17 18:05:12.004415746 +0000 UTC m=+1823.095722511"
Sep 17 18:05:12.504650 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:12.504620    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"
Sep 17 18:05:17.504074 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:17.504010    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"
Sep 17 18:05:17.516467 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:17.516441    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"
Sep 17 18:05:18.018323 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:18.018293    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"
Sep 17 18:05:40.972677 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:05:40.972638    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll"
Sep 17 18:06:12.112501 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.112413    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"]
Sep 17 18:06:12.115800 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.115776    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"
Sep 17 18:06:12.118784 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.118762    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:06:12.118898 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.118836    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Sep 17 18:06:12.118898 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.118864    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 17 18:06:12.118993 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.118912    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 17 18:06:12.119037 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.119021    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-k5b8m\""
Sep 17 18:06:12.119629 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.119613    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 17 18:06:12.134784 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.134761    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"]
Sep 17 18:06:12.181654 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.181615    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-model-cache\") pod \"auth-invalid-token-test-kserve-d78cb44d9-q9wm4\" (UID: \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"
Sep 17 18:06:12.181817 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.181686    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-home\") pod \"auth-invalid-token-test-kserve-d78cb44d9-q9wm4\" (UID: \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"
Sep 17 18:06:12.181817 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.181758    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-d78cb44d9-q9wm4\" (UID: \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"
Sep 17 18:06:12.181817 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.181792    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-dshm\") pod \"auth-invalid-token-test-kserve-d78cb44d9-q9wm4\" (UID: \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"
Sep 17 18:06:12.181953 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.181815    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-tls-certs\") pod \"auth-invalid-token-test-kserve-d78cb44d9-q9wm4\" (UID: \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"
Sep 17 18:06:12.181953 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.181844    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-cabundle-cert\") pod \"auth-invalid-token-test-kserve-d78cb44d9-q9wm4\" (UID: \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"
Sep 17 18:06:12.181953 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.181899    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jvbk\" (UniqueName: \"kubernetes.io/projected/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-kube-api-access-8jvbk\") pod \"auth-invalid-token-test-kserve-d78cb44d9-q9wm4\" (UID: \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"
Sep 17 18:06:12.181953 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.181950    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-tmp-dir\") pod \"auth-invalid-token-test-kserve-d78cb44d9-q9wm4\" (UID: \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"
Sep 17 18:06:12.282964 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.282941    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-tmp-dir\") pod \"auth-invalid-token-test-kserve-d78cb44d9-q9wm4\" (UID: \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"
Sep 17 18:06:12.283103 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.282975    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-model-cache\") pod \"auth-invalid-token-test-kserve-d78cb44d9-q9wm4\" (UID: \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"
Sep 17 18:06:12.283103 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.283001    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-home\") pod \"auth-invalid-token-test-kserve-d78cb44d9-q9wm4\" (UID: \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"
Sep 17 18:06:12.283103 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.283031    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-d78cb44d9-q9wm4\" (UID: \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"
Sep 17 18:06:12.283223 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.283140    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-dshm\") pod \"auth-invalid-token-test-kserve-d78cb44d9-q9wm4\" (UID: \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"
Sep 17 18:06:12.283223 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.283173    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-tls-certs\") pod \"auth-invalid-token-test-kserve-d78cb44d9-q9wm4\" (UID: \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"
Sep 17 18:06:12.283223 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.283195    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-cabundle-cert\") pod \"auth-invalid-token-test-kserve-d78cb44d9-q9wm4\" (UID: \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"
Sep 17 18:06:12.283360 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.283241    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jvbk\" (UniqueName: \"kubernetes.io/projected/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-kube-api-access-8jvbk\") pod \"auth-invalid-token-test-kserve-d78cb44d9-q9wm4\" (UID: \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"
Sep 17 18:06:12.283421 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.283397    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-model-cache\") pod \"auth-invalid-token-test-kserve-d78cb44d9-q9wm4\" (UID: \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"
Sep 17 18:06:12.283473 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.283451    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-home\") pod \"auth-invalid-token-test-kserve-d78cb44d9-q9wm4\" (UID: \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"
Sep 17 18:06:12.283528 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.283480    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-tmp-dir\") pod \"auth-invalid-token-test-kserve-d78cb44d9-q9wm4\" (UID: \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"
Sep 17 18:06:12.283528 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.283518    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-d78cb44d9-q9wm4\" (UID: \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"
Sep 17 18:06:12.283921 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.283901    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-cabundle-cert\") pod \"auth-invalid-token-test-kserve-d78cb44d9-q9wm4\" (UID: \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"
Sep 17 18:06:12.285357 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.285338    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-dshm\") pod \"auth-invalid-token-test-kserve-d78cb44d9-q9wm4\" (UID: \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"
Sep 17 18:06:12.285774 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.285756    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-tls-certs\") pod \"auth-invalid-token-test-kserve-d78cb44d9-q9wm4\" (UID: \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"
Sep 17 18:06:12.296106 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.296078    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jvbk\" (UniqueName: \"kubernetes.io/projected/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-kube-api-access-8jvbk\") pod \"auth-invalid-token-test-kserve-d78cb44d9-q9wm4\" (UID: \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"
Sep 17 18:06:12.426013 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.425937    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"
Sep 17 18:06:12.724579 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:12.724507    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"]
Sep 17 18:06:12.727838 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:06:12.727808    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a7152b8_5386_45b2_b110_7dea7bd9ba4f.slice/crio-cc5d9b01574fba966950497c8547d2baa1fd540e1fd562f6b399df7b8fb2c86d WatchSource:0}: Error finding container cc5d9b01574fba966950497c8547d2baa1fd540e1fd562f6b399df7b8fb2c86d: Status 404 returned error can't find the container with id cc5d9b01574fba966950497c8547d2baa1fd540e1fd562f6b399df7b8fb2c86d
Sep 17 18:06:13.243581 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:13.243548    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4" event={"ID":"7a7152b8-5386-45b2-b110-7dea7bd9ba4f","Type":"ContainerStarted","Data":"380fc88454acfc32ba8d17244418d8066e47959fbfc8c1ac12174382f5c81f0b"}
Sep 17 18:06:13.244025 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:13.243602    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4" event={"ID":"7a7152b8-5386-45b2-b110-7dea7bd9ba4f","Type":"ContainerStarted","Data":"cc5d9b01574fba966950497c8547d2baa1fd540e1fd562f6b399df7b8fb2c86d"}
Sep 17 18:06:15.254568 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:15.254478    2583 generic.go:356] "Generic (PLEG): container finished" podID="7a7152b8-5386-45b2-b110-7dea7bd9ba4f" containerID="380fc88454acfc32ba8d17244418d8066e47959fbfc8c1ac12174382f5c81f0b" exitCode=0
Sep 17 18:06:15.254568 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:15.254553    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4" event={"ID":"7a7152b8-5386-45b2-b110-7dea7bd9ba4f","Type":"ContainerDied","Data":"380fc88454acfc32ba8d17244418d8066e47959fbfc8c1ac12174382f5c81f0b"}
Sep 17 18:06:15.256072 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:15.256055    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:06:31.377714 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:31.377654    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll"]
Sep 17 18:06:31.385760 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:31.385736    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"]
Sep 17 18:06:31.397154 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:06:31.397076    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:06:31.397154 ip-10-0-141-30 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:06:31.397154 ip-10-0-141-30 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 18:06:31.397154 ip-10-0-141-30 kubenswrapper[2583]:  > containerID="bdd2e9aad6ceb87727c105c734ead2323c44dc6646bd809257bc85dd0cb2c33a" cmd=["/bin/sleep","15"]
Sep 17 18:06:31.397154 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:31.397143    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr" message="" err=<
Sep 17 18:06:31.397154 ip-10-0-141-30 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:06:31.397154 ip-10-0-141-30 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 18:06:31.397154 ip-10-0-141-30 kubenswrapper[2583]:  >
Sep 17 18:06:31.397636 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:06:31.397161    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:06:31.397636 ip-10-0-141-30 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:06:31.397636 ip-10-0-141-30 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 18:06:31.397636 ip-10-0-141-30 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr" podUID="5e610c8d-5c8b-474f-b597-1e0e81c4b224" containerName="main" containerID="cri-o://bdd2e9aad6ceb87727c105c734ead2323c44dc6646bd809257bc85dd0cb2c33a"
Sep 17 18:06:31.397636 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:31.397212    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr" podUID="5e610c8d-5c8b-474f-b597-1e0e81c4b224" containerName="main" containerID="cri-o://bdd2e9aad6ceb87727c105c734ead2323c44dc6646bd809257bc85dd0cb2c33a" gracePeriod=60
Sep 17 18:06:32.008148 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:32.008108    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr" podUID="5e610c8d-5c8b-474f-b597-1e0e81c4b224" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 17 18:06:33.007289 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:33.007250    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr" podUID="5e610c8d-5c8b-474f-b597-1e0e81c4b224" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 17 18:06:33.007806 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:33.007371    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"
Sep 17 18:06:34.008125 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:34.008077    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr" podUID="5e610c8d-5c8b-474f-b597-1e0e81c4b224" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 17 18:06:35.007679 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:35.007612    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr" podUID="5e610c8d-5c8b-474f-b597-1e0e81c4b224" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 17 18:06:36.008065 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:36.008013    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr" podUID="5e610c8d-5c8b-474f-b597-1e0e81c4b224" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 17 18:06:37.007634 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:37.007580    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr" podUID="5e610c8d-5c8b-474f-b597-1e0e81c4b224" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 17 18:06:38.007637 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:38.007581    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr" podUID="5e610c8d-5c8b-474f-b597-1e0e81c4b224" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 17 18:06:39.007199 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:39.007149    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr" podUID="5e610c8d-5c8b-474f-b597-1e0e81c4b224" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 17 18:06:40.007799 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:40.007744    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr" podUID="5e610c8d-5c8b-474f-b597-1e0e81c4b224" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 17 18:06:40.973190 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:40.973152    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll" podUID="45b08415-30f1-4a74-bdf5-56292806f668" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:06:41.007943 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:41.007900    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr" podUID="5e610c8d-5c8b-474f-b597-1e0e81c4b224" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 17 18:06:42.008110 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:42.008069    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr" podUID="5e610c8d-5c8b-474f-b597-1e0e81c4b224" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 17 18:06:42.397716 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:42.397679    2583 generic.go:356] "Generic (PLEG): container finished" podID="5e610c8d-5c8b-474f-b597-1e0e81c4b224" containerID="bdd2e9aad6ceb87727c105c734ead2323c44dc6646bd809257bc85dd0cb2c33a" exitCode=0
Sep 17 18:06:42.397716 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:42.397702    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr" event={"ID":"5e610c8d-5c8b-474f-b597-1e0e81c4b224","Type":"ContainerDied","Data":"bdd2e9aad6ceb87727c105c734ead2323c44dc6646bd809257bc85dd0cb2c33a"}
Sep 17 18:06:43.773557 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:43.773531    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"
Sep 17 18:06:43.874365 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:43.874334    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hd4tx\" (UniqueName: \"kubernetes.io/projected/5e610c8d-5c8b-474f-b597-1e0e81c4b224-kube-api-access-hd4tx\") pod \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\" (UID: \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\") "
Sep 17 18:06:43.874529 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:43.874388    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e610c8d-5c8b-474f-b597-1e0e81c4b224-model-cache\") pod \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\" (UID: \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\") "
Sep 17 18:06:43.874529 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:43.874427    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e610c8d-5c8b-474f-b597-1e0e81c4b224-kserve-provision-location\") pod \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\" (UID: \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\") "
Sep 17 18:06:43.874529 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:43.874474    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e610c8d-5c8b-474f-b597-1e0e81c4b224-tmp-dir\") pod \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\" (UID: \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\") "
Sep 17 18:06:43.874529 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:43.874490    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e610c8d-5c8b-474f-b597-1e0e81c4b224-home\") pod \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\" (UID: \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\") "
Sep 17 18:06:43.874529 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:43.874507    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e610c8d-5c8b-474f-b597-1e0e81c4b224-tls-certs\") pod \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\" (UID: \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\") "
Sep 17 18:06:43.875212 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:43.874535    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e610c8d-5c8b-474f-b597-1e0e81c4b224-cabundle-cert\") pod \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\" (UID: \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\") "
Sep 17 18:06:43.875212 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:43.874559    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e610c8d-5c8b-474f-b597-1e0e81c4b224-dshm\") pod \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\" (UID: \"5e610c8d-5c8b-474f-b597-1e0e81c4b224\") "
Sep 17 18:06:43.875212 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:43.874717    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e610c8d-5c8b-474f-b597-1e0e81c4b224-model-cache" (OuterVolumeSpecName: "model-cache") pod "5e610c8d-5c8b-474f-b597-1e0e81c4b224" (UID: "5e610c8d-5c8b-474f-b597-1e0e81c4b224"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:06:43.875212 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:43.874770    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e610c8d-5c8b-474f-b597-1e0e81c4b224-home" (OuterVolumeSpecName: "home") pod "5e610c8d-5c8b-474f-b597-1e0e81c4b224" (UID: "5e610c8d-5c8b-474f-b597-1e0e81c4b224"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:06:43.875212 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:43.874874    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e610c8d-5c8b-474f-b597-1e0e81c4b224-model-cache\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:06:43.875212 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:43.874893    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e610c8d-5c8b-474f-b597-1e0e81c4b224-home\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:06:43.875212 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:43.875009    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e610c8d-5c8b-474f-b597-1e0e81c4b224-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5e610c8d-5c8b-474f-b597-1e0e81c4b224" (UID: "5e610c8d-5c8b-474f-b597-1e0e81c4b224"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:06:43.875212 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:43.875085    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e610c8d-5c8b-474f-b597-1e0e81c4b224-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5e610c8d-5c8b-474f-b597-1e0e81c4b224" (UID: "5e610c8d-5c8b-474f-b597-1e0e81c4b224"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:06:43.876832 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:43.876807    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e610c8d-5c8b-474f-b597-1e0e81c4b224-dshm" (OuterVolumeSpecName: "dshm") pod "5e610c8d-5c8b-474f-b597-1e0e81c4b224" (UID: "5e610c8d-5c8b-474f-b597-1e0e81c4b224"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:06:43.877022 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:43.876998    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e610c8d-5c8b-474f-b597-1e0e81c4b224-kube-api-access-hd4tx" (OuterVolumeSpecName: "kube-api-access-hd4tx") pod "5e610c8d-5c8b-474f-b597-1e0e81c4b224" (UID: "5e610c8d-5c8b-474f-b597-1e0e81c4b224"). InnerVolumeSpecName "kube-api-access-hd4tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:06:43.877330 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:43.877302    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e610c8d-5c8b-474f-b597-1e0e81c4b224-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5e610c8d-5c8b-474f-b597-1e0e81c4b224" (UID: "5e610c8d-5c8b-474f-b597-1e0e81c4b224"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:06:43.891717 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:43.891647    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e610c8d-5c8b-474f-b597-1e0e81c4b224-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5e610c8d-5c8b-474f-b597-1e0e81c4b224" (UID: "5e610c8d-5c8b-474f-b597-1e0e81c4b224"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:06:43.975423 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:43.975400    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e610c8d-5c8b-474f-b597-1e0e81c4b224-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:06:43.975423 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:43.975421    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e610c8d-5c8b-474f-b597-1e0e81c4b224-cabundle-cert\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:06:43.975551 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:43.975429    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e610c8d-5c8b-474f-b597-1e0e81c4b224-dshm\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:06:43.975551 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:43.975438    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd4tx\" (UniqueName: \"kubernetes.io/projected/5e610c8d-5c8b-474f-b597-1e0e81c4b224-kube-api-access-hd4tx\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:06:43.975551 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:43.975447    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e610c8d-5c8b-474f-b597-1e0e81c4b224-kserve-provision-location\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:06:43.975551 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:43.975474    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e610c8d-5c8b-474f-b597-1e0e81c4b224-tmp-dir\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:06:44.008351 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:44.008269    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr" podUID="5e610c8d-5c8b-474f-b597-1e0e81c4b224" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": context deadline exceeded"
Sep 17 18:06:44.412514 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:44.412472    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4" event={"ID":"7a7152b8-5386-45b2-b110-7dea7bd9ba4f","Type":"ContainerStarted","Data":"b0b448b9a74f3280eb58f5eadd6f4f168be7d4ab6e504cff95583887e451608d"}
Sep 17 18:06:44.414094 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:44.414069    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr" event={"ID":"5e610c8d-5c8b-474f-b597-1e0e81c4b224","Type":"ContainerDied","Data":"3b588b253b8e17d60b9092f5e7dd2b18fd2f4a5769bcdbe67ec8836339f31662"}
Sep 17 18:06:44.414094 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:44.414088    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"
Sep 17 18:06:44.414306 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:44.414126    2583 scope.go:117] "RemoveContainer" containerID="bdd2e9aad6ceb87727c105c734ead2323c44dc6646bd809257bc85dd0cb2c33a"
Sep 17 18:06:44.422932 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:44.422917    2583 scope.go:117] "RemoveContainer" containerID="47203a33169e1ab93a59f2448fa2bb6b0f32acd74ccd252ec6e4696e24ba0b22"
Sep 17 18:06:44.426713 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:44.426691    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"
Sep 17 18:06:44.448982 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:44.448878    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4" podStartSLOduration=1.471959128 podStartE2EDuration="32.448860843s" totalImagesPullingTime="28.832838095s" totalInitContainerRuntime="2.14406362s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:06:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:06:44.447498077 +0000 UTC m=+1915.538804872" watchObservedRunningTime="2026-09-17 18:06:44.448860843 +0000 UTC m=+1915.540167609"
Sep 17 18:06:44.461803 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:44.461777    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"]
Sep 17 18:06:44.466156 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:44.466132    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5b769465b9-l6nzr"]
Sep 17 18:06:45.041347 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:45.041315    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e610c8d-5c8b-474f-b597-1e0e81c4b224" path="/var/lib/kubelet/pods/5e610c8d-5c8b-474f-b597-1e0e81c4b224/volumes"
Sep 17 18:06:46.404458 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:46.404414    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll" podUID="45b08415-30f1-4a74-bdf5-56292806f668" containerName="main" containerID="cri-o://30a5f51c710a60b145add1e62be9bbb385274232ecbdbdafe73b67fd420f1c90" gracePeriod=45
Sep 17 18:06:50.972853 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:50.972809    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll" podUID="45b08415-30f1-4a74-bdf5-56292806f668" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:06:52.426405 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:52.426372    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"
Sep 17 18:06:52.427769 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:06:52.427737    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4" podUID="7a7152b8-5386-45b2-b110-7dea7bd9ba4f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 17 18:07:00.972627 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:00.972587    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll" podUID="45b08415-30f1-4a74-bdf5-56292806f668" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:07:00.973195 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:00.972794    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll"
Sep 17 18:07:02.426799 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:02.426760    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4" podUID="7a7152b8-5386-45b2-b110-7dea7bd9ba4f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 17 18:07:10.973363 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:10.973325    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll" podUID="45b08415-30f1-4a74-bdf5-56292806f668" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:07:12.427267 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:12.427226    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4" podUID="7a7152b8-5386-45b2-b110-7dea7bd9ba4f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 17 18:07:16.558715 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:16.558680    2583 generic.go:356] "Generic (PLEG): container finished" podID="45b08415-30f1-4a74-bdf5-56292806f668" containerID="30a5f51c710a60b145add1e62be9bbb385274232ecbdbdafe73b67fd420f1c90" exitCode=0
Sep 17 18:07:16.559096 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:16.558757    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll" event={"ID":"45b08415-30f1-4a74-bdf5-56292806f668","Type":"ContainerDied","Data":"30a5f51c710a60b145add1e62be9bbb385274232ecbdbdafe73b67fd420f1c90"}
Sep 17 18:07:16.647504 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:16.647474    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll"
Sep 17 18:07:16.661349 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:16.661326    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b08415-30f1-4a74-bdf5-56292806f668-tls-certs\") pod \"45b08415-30f1-4a74-bdf5-56292806f668\" (UID: \"45b08415-30f1-4a74-bdf5-56292806f668\") "
Sep 17 18:07:16.661424 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:16.661382    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbcw9\" (UniqueName: \"kubernetes.io/projected/45b08415-30f1-4a74-bdf5-56292806f668-kube-api-access-qbcw9\") pod \"45b08415-30f1-4a74-bdf5-56292806f668\" (UID: \"45b08415-30f1-4a74-bdf5-56292806f668\") "
Sep 17 18:07:16.663296 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:16.663238    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45b08415-30f1-4a74-bdf5-56292806f668-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "45b08415-30f1-4a74-bdf5-56292806f668" (UID: "45b08415-30f1-4a74-bdf5-56292806f668"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:07:16.663421 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:16.663336    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45b08415-30f1-4a74-bdf5-56292806f668-kube-api-access-qbcw9" (OuterVolumeSpecName: "kube-api-access-qbcw9") pod "45b08415-30f1-4a74-bdf5-56292806f668" (UID: "45b08415-30f1-4a74-bdf5-56292806f668"). InnerVolumeSpecName "kube-api-access-qbcw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:07:16.762334 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:16.762306    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45b08415-30f1-4a74-bdf5-56292806f668-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:07:16.762334 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:16.762332    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbcw9\" (UniqueName: \"kubernetes.io/projected/45b08415-30f1-4a74-bdf5-56292806f668-kube-api-access-qbcw9\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:07:17.564007 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:17.563976    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll"
Sep 17 18:07:17.564443 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:17.563981    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll" event={"ID":"45b08415-30f1-4a74-bdf5-56292806f668","Type":"ContainerDied","Data":"8d90164f11029e0fb4c50a935fd080257d81ad622b1baac10225e4d54d6b6c74"}
Sep 17 18:07:17.564443 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:17.564109    2583 scope.go:117] "RemoveContainer" containerID="30a5f51c710a60b145add1e62be9bbb385274232ecbdbdafe73b67fd420f1c90"
Sep 17 18:07:17.582999 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:17.582976    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll"]
Sep 17 18:07:17.587353 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:17.587330    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69rpqll"]
Sep 17 18:07:19.039993 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:19.039961    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45b08415-30f1-4a74-bdf5-56292806f668" path="/var/lib/kubelet/pods/45b08415-30f1-4a74-bdf5-56292806f668/volumes"
Sep 17 18:07:22.426653 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:22.426606    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4" podUID="7a7152b8-5386-45b2-b110-7dea7bd9ba4f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 17 18:07:26.537258 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.537222    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"]
Sep 17 18:07:26.537676 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.537619    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e610c8d-5c8b-474f-b597-1e0e81c4b224" containerName="storage-initializer"
Sep 17 18:07:26.537676 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.537632    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e610c8d-5c8b-474f-b597-1e0e81c4b224" containerName="storage-initializer"
Sep 17 18:07:26.537676 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.537649    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45b08415-30f1-4a74-bdf5-56292806f668" containerName="main"
Sep 17 18:07:26.537676 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.537674    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="45b08415-30f1-4a74-bdf5-56292806f668" containerName="main"
Sep 17 18:07:26.537843 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.537684    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e610c8d-5c8b-474f-b597-1e0e81c4b224" containerName="main"
Sep 17 18:07:26.537843 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.537692    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e610c8d-5c8b-474f-b597-1e0e81c4b224" containerName="main"
Sep 17 18:07:26.537843 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.537744    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="45b08415-30f1-4a74-bdf5-56292806f668" containerName="main"
Sep 17 18:07:26.537843 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.537751    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e610c8d-5c8b-474f-b597-1e0e81c4b224" containerName="main"
Sep 17 18:07:26.540195 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.540174    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"
Sep 17 18:07:26.542951 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.542923    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 17 18:07:26.543089 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.542975    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-fsg6t\""
Sep 17 18:07:26.543089 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.542998    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:07:26.544385 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.544361    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 17 18:07:26.544501 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.544430    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 17 18:07:26.544671 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.544643    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 17 18:07:26.552614 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.552595    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"]
Sep 17 18:07:26.638704 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.638652    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec262e14-b98c-42c5-b319-1d988833a4b9-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-85f864cf88-pr5gr\" (UID: \"ec262e14-b98c-42c5-b319-1d988833a4b9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"
Sep 17 18:07:26.638870 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.638746    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8stb\" (UniqueName: \"kubernetes.io/projected/ec262e14-b98c-42c5-b319-1d988833a4b9-kube-api-access-n8stb\") pod \"tokenizer-migration-path-test-kserve-85f864cf88-pr5gr\" (UID: \"ec262e14-b98c-42c5-b319-1d988833a4b9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"
Sep 17 18:07:26.638870 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.638794    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec262e14-b98c-42c5-b319-1d988833a4b9-tls-certs\") pod \"tokenizer-migration-path-test-kserve-85f864cf88-pr5gr\" (UID: \"ec262e14-b98c-42c5-b319-1d988833a4b9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"
Sep 17 18:07:26.638870 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.638838    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec262e14-b98c-42c5-b319-1d988833a4b9-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-85f864cf88-pr5gr\" (UID: \"ec262e14-b98c-42c5-b319-1d988833a4b9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"
Sep 17 18:07:26.638987 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.638897    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec262e14-b98c-42c5-b319-1d988833a4b9-model-cache\") pod \"tokenizer-migration-path-test-kserve-85f864cf88-pr5gr\" (UID: \"ec262e14-b98c-42c5-b319-1d988833a4b9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"
Sep 17 18:07:26.638987 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.638927    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec262e14-b98c-42c5-b319-1d988833a4b9-dshm\") pod \"tokenizer-migration-path-test-kserve-85f864cf88-pr5gr\" (UID: \"ec262e14-b98c-42c5-b319-1d988833a4b9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"
Sep 17 18:07:26.639059 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.639031    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec262e14-b98c-42c5-b319-1d988833a4b9-home\") pod \"tokenizer-migration-path-test-kserve-85f864cf88-pr5gr\" (UID: \"ec262e14-b98c-42c5-b319-1d988833a4b9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"
Sep 17 18:07:26.639094 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.639071    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec262e14-b98c-42c5-b319-1d988833a4b9-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-85f864cf88-pr5gr\" (UID: \"ec262e14-b98c-42c5-b319-1d988833a4b9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"
Sep 17 18:07:26.739481 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.739446    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec262e14-b98c-42c5-b319-1d988833a4b9-home\") pod \"tokenizer-migration-path-test-kserve-85f864cf88-pr5gr\" (UID: \"ec262e14-b98c-42c5-b319-1d988833a4b9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"
Sep 17 18:07:26.739481 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.739484    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec262e14-b98c-42c5-b319-1d988833a4b9-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-85f864cf88-pr5gr\" (UID: \"ec262e14-b98c-42c5-b319-1d988833a4b9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"
Sep 17 18:07:26.739691 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.739506    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec262e14-b98c-42c5-b319-1d988833a4b9-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-85f864cf88-pr5gr\" (UID: \"ec262e14-b98c-42c5-b319-1d988833a4b9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"
Sep 17 18:07:26.739691 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.739555    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8stb\" (UniqueName: \"kubernetes.io/projected/ec262e14-b98c-42c5-b319-1d988833a4b9-kube-api-access-n8stb\") pod \"tokenizer-migration-path-test-kserve-85f864cf88-pr5gr\" (UID: \"ec262e14-b98c-42c5-b319-1d988833a4b9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"
Sep 17 18:07:26.739691 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.739580    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec262e14-b98c-42c5-b319-1d988833a4b9-tls-certs\") pod \"tokenizer-migration-path-test-kserve-85f864cf88-pr5gr\" (UID: \"ec262e14-b98c-42c5-b319-1d988833a4b9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"
Sep 17 18:07:26.739691 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.739625    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec262e14-b98c-42c5-b319-1d988833a4b9-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-85f864cf88-pr5gr\" (UID: \"ec262e14-b98c-42c5-b319-1d988833a4b9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"
Sep 17 18:07:26.739691 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.739674    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec262e14-b98c-42c5-b319-1d988833a4b9-model-cache\") pod \"tokenizer-migration-path-test-kserve-85f864cf88-pr5gr\" (UID: \"ec262e14-b98c-42c5-b319-1d988833a4b9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"
Sep 17 18:07:26.739953 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.739711    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec262e14-b98c-42c5-b319-1d988833a4b9-dshm\") pod \"tokenizer-migration-path-test-kserve-85f864cf88-pr5gr\" (UID: \"ec262e14-b98c-42c5-b319-1d988833a4b9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"
Sep 17 18:07:26.739953 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.739917    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec262e14-b98c-42c5-b319-1d988833a4b9-home\") pod \"tokenizer-migration-path-test-kserve-85f864cf88-pr5gr\" (UID: \"ec262e14-b98c-42c5-b319-1d988833a4b9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"
Sep 17 18:07:26.739953 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.739941    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec262e14-b98c-42c5-b319-1d988833a4b9-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-85f864cf88-pr5gr\" (UID: \"ec262e14-b98c-42c5-b319-1d988833a4b9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"
Sep 17 18:07:26.740105 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.740034    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec262e14-b98c-42c5-b319-1d988833a4b9-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-85f864cf88-pr5gr\" (UID: \"ec262e14-b98c-42c5-b319-1d988833a4b9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"
Sep 17 18:07:26.740192 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.740169    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec262e14-b98c-42c5-b319-1d988833a4b9-model-cache\") pod \"tokenizer-migration-path-test-kserve-85f864cf88-pr5gr\" (UID: \"ec262e14-b98c-42c5-b319-1d988833a4b9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"
Sep 17 18:07:26.740459 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.740439    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec262e14-b98c-42c5-b319-1d988833a4b9-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-85f864cf88-pr5gr\" (UID: \"ec262e14-b98c-42c5-b319-1d988833a4b9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"
Sep 17 18:07:26.741916 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.741890    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec262e14-b98c-42c5-b319-1d988833a4b9-dshm\") pod \"tokenizer-migration-path-test-kserve-85f864cf88-pr5gr\" (UID: \"ec262e14-b98c-42c5-b319-1d988833a4b9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"
Sep 17 18:07:26.742192 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.742174    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec262e14-b98c-42c5-b319-1d988833a4b9-tls-certs\") pod \"tokenizer-migration-path-test-kserve-85f864cf88-pr5gr\" (UID: \"ec262e14-b98c-42c5-b319-1d988833a4b9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"
Sep 17 18:07:26.747072 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.747047    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8stb\" (UniqueName: \"kubernetes.io/projected/ec262e14-b98c-42c5-b319-1d988833a4b9-kube-api-access-n8stb\") pod \"tokenizer-migration-path-test-kserve-85f864cf88-pr5gr\" (UID: \"ec262e14-b98c-42c5-b319-1d988833a4b9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"
Sep 17 18:07:26.850349 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:26.850317    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"
Sep 17 18:07:27.133960 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:27.133886    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"]
Sep 17 18:07:27.137222 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:07:27.137196    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec262e14_b98c_42c5_b319_1d988833a4b9.slice/crio-b28f69c050833d0f658644bdeb3a61901cc89b4696800362949e4172ce746163 WatchSource:0}: Error finding container b28f69c050833d0f658644bdeb3a61901cc89b4696800362949e4172ce746163: Status 404 returned error can't find the container with id b28f69c050833d0f658644bdeb3a61901cc89b4696800362949e4172ce746163
Sep 17 18:07:27.614770 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:27.614730    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr" event={"ID":"ec262e14-b98c-42c5-b319-1d988833a4b9","Type":"ContainerStarted","Data":"52600df4746900e42c445369ac9349f86bdea059de8db76cae8352fb5989d364"}
Sep 17 18:07:27.615265 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:27.614815    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr" event={"ID":"ec262e14-b98c-42c5-b319-1d988833a4b9","Type":"ContainerStarted","Data":"b28f69c050833d0f658644bdeb3a61901cc89b4696800362949e4172ce746163"}
Sep 17 18:07:29.625692 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:29.625644    2583 generic.go:356] "Generic (PLEG): container finished" podID="ec262e14-b98c-42c5-b319-1d988833a4b9" containerID="52600df4746900e42c445369ac9349f86bdea059de8db76cae8352fb5989d364" exitCode=0
Sep 17 18:07:29.626071 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:29.625717    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr" event={"ID":"ec262e14-b98c-42c5-b319-1d988833a4b9","Type":"ContainerDied","Data":"52600df4746900e42c445369ac9349f86bdea059de8db76cae8352fb5989d364"}
Sep 17 18:07:30.631834 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:30.631799    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr" event={"ID":"ec262e14-b98c-42c5-b319-1d988833a4b9","Type":"ContainerStarted","Data":"dca812049211351f323b6d3caf778e3309b1cb8c290564fab927d45c85eff865"}
Sep 17 18:07:30.655443 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:30.655388    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr" podStartSLOduration=2.388338385 podStartE2EDuration="4.655361143s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.267022758s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:07:30.654679438 +0000 UTC m=+1961.745986222" watchObservedRunningTime="2026-09-17 18:07:30.655361143 +0000 UTC m=+1961.746667909"
Sep 17 18:07:30.851085 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:30.851051    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"
Sep 17 18:07:32.427286 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:32.427188    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4" podUID="7a7152b8-5386-45b2-b110-7dea7bd9ba4f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 17 18:07:36.850743 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:36.850704    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"
Sep 17 18:07:36.869328 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:36.869292    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"
Sep 17 18:07:37.675028 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:37.674990    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"
Sep 17 18:07:42.426613 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:42.426568    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4" podUID="7a7152b8-5386-45b2-b110-7dea7bd9ba4f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 17 18:07:52.426712 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:07:52.426649    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4" podUID="7a7152b8-5386-45b2-b110-7dea7bd9ba4f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 17 18:08:02.427176 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:02.427125    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4" podUID="7a7152b8-5386-45b2-b110-7dea7bd9ba4f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 17 18:08:04.382380 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.382346    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"]
Sep 17 18:08:04.391583 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:08:04.391545    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:08:04.391583 ip-10-0-141-30 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:08:04.391583 ip-10-0-141-30 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 18:08:04.391583 ip-10-0-141-30 kubenswrapper[2583]:  > containerID="dca812049211351f323b6d3caf778e3309b1cb8c290564fab927d45c85eff865" cmd=["/bin/sleep","15"]
Sep 17 18:08:04.391903 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.391604    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr" message="" err=<
Sep 17 18:08:04.391903 ip-10-0-141-30 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:08:04.391903 ip-10-0-141-30 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 18:08:04.391903 ip-10-0-141-30 kubenswrapper[2583]:  >
Sep 17 18:08:04.391903 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:08:04.391620    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:08:04.391903 ip-10-0-141-30 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:08:04.391903 ip-10-0-141-30 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 18:08:04.391903 ip-10-0-141-30 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr" podUID="ec262e14-b98c-42c5-b319-1d988833a4b9" containerName="main" containerID="cri-o://dca812049211351f323b6d3caf778e3309b1cb8c290564fab927d45c85eff865"
Sep 17 18:08:04.391903 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.391688    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr" podUID="ec262e14-b98c-42c5-b319-1d988833a4b9" containerName="main" containerID="cri-o://dca812049211351f323b6d3caf778e3309b1cb8c290564fab927d45c85eff865" gracePeriod=60
Sep 17 18:08:04.731443 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.731411    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"
Sep 17 18:08:04.782902 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.782865    2583 generic.go:356] "Generic (PLEG): container finished" podID="ec262e14-b98c-42c5-b319-1d988833a4b9" containerID="dca812049211351f323b6d3caf778e3309b1cb8c290564fab927d45c85eff865" exitCode=0
Sep 17 18:08:04.783065 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.782925    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"
Sep 17 18:08:04.783065 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.782947    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr" event={"ID":"ec262e14-b98c-42c5-b319-1d988833a4b9","Type":"ContainerDied","Data":"dca812049211351f323b6d3caf778e3309b1cb8c290564fab927d45c85eff865"}
Sep 17 18:08:04.783065 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.783018    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr" event={"ID":"ec262e14-b98c-42c5-b319-1d988833a4b9","Type":"ContainerDied","Data":"b28f69c050833d0f658644bdeb3a61901cc89b4696800362949e4172ce746163"}
Sep 17 18:08:04.783065 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.783053    2583 scope.go:117] "RemoveContainer" containerID="dca812049211351f323b6d3caf778e3309b1cb8c290564fab927d45c85eff865"
Sep 17 18:08:04.791508 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.791489    2583 scope.go:117] "RemoveContainer" containerID="52600df4746900e42c445369ac9349f86bdea059de8db76cae8352fb5989d364"
Sep 17 18:08:04.798747 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.798732    2583 scope.go:117] "RemoveContainer" containerID="dca812049211351f323b6d3caf778e3309b1cb8c290564fab927d45c85eff865"
Sep 17 18:08:04.798995 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:08:04.798979    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dca812049211351f323b6d3caf778e3309b1cb8c290564fab927d45c85eff865\": container with ID starting with dca812049211351f323b6d3caf778e3309b1cb8c290564fab927d45c85eff865 not found: ID does not exist" containerID="dca812049211351f323b6d3caf778e3309b1cb8c290564fab927d45c85eff865"
Sep 17 18:08:04.799039 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.799005    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dca812049211351f323b6d3caf778e3309b1cb8c290564fab927d45c85eff865"} err="failed to get container status \"dca812049211351f323b6d3caf778e3309b1cb8c290564fab927d45c85eff865\": rpc error: code = NotFound desc = could not find container \"dca812049211351f323b6d3caf778e3309b1cb8c290564fab927d45c85eff865\": container with ID starting with dca812049211351f323b6d3caf778e3309b1cb8c290564fab927d45c85eff865 not found: ID does not exist"
Sep 17 18:08:04.799039 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.799021    2583 scope.go:117] "RemoveContainer" containerID="52600df4746900e42c445369ac9349f86bdea059de8db76cae8352fb5989d364"
Sep 17 18:08:04.799301 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:08:04.799279    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52600df4746900e42c445369ac9349f86bdea059de8db76cae8352fb5989d364\": container with ID starting with 52600df4746900e42c445369ac9349f86bdea059de8db76cae8352fb5989d364 not found: ID does not exist" containerID="52600df4746900e42c445369ac9349f86bdea059de8db76cae8352fb5989d364"
Sep 17 18:08:04.799353 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.799310    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52600df4746900e42c445369ac9349f86bdea059de8db76cae8352fb5989d364"} err="failed to get container status \"52600df4746900e42c445369ac9349f86bdea059de8db76cae8352fb5989d364\": rpc error: code = NotFound desc = could not find container \"52600df4746900e42c445369ac9349f86bdea059de8db76cae8352fb5989d364\": container with ID starting with 52600df4746900e42c445369ac9349f86bdea059de8db76cae8352fb5989d364 not found: ID does not exist"
Sep 17 18:08:04.880316 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.880284    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec262e14-b98c-42c5-b319-1d988833a4b9-dshm\") pod \"ec262e14-b98c-42c5-b319-1d988833a4b9\" (UID: \"ec262e14-b98c-42c5-b319-1d988833a4b9\") "
Sep 17 18:08:04.880486 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.880331    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec262e14-b98c-42c5-b319-1d988833a4b9-home\") pod \"ec262e14-b98c-42c5-b319-1d988833a4b9\" (UID: \"ec262e14-b98c-42c5-b319-1d988833a4b9\") "
Sep 17 18:08:04.880486 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.880372    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec262e14-b98c-42c5-b319-1d988833a4b9-tmp-dir\") pod \"ec262e14-b98c-42c5-b319-1d988833a4b9\" (UID: \"ec262e14-b98c-42c5-b319-1d988833a4b9\") "
Sep 17 18:08:04.880486 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.880395    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec262e14-b98c-42c5-b319-1d988833a4b9-model-cache\") pod \"ec262e14-b98c-42c5-b319-1d988833a4b9\" (UID: \"ec262e14-b98c-42c5-b319-1d988833a4b9\") "
Sep 17 18:08:04.880486 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.880413    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec262e14-b98c-42c5-b319-1d988833a4b9-kserve-provision-location\") pod \"ec262e14-b98c-42c5-b319-1d988833a4b9\" (UID: \"ec262e14-b98c-42c5-b319-1d988833a4b9\") "
Sep 17 18:08:04.880486 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.880439    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec262e14-b98c-42c5-b319-1d988833a4b9-tls-certs\") pod \"ec262e14-b98c-42c5-b319-1d988833a4b9\" (UID: \"ec262e14-b98c-42c5-b319-1d988833a4b9\") "
Sep 17 18:08:04.880486 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.880469    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8stb\" (UniqueName: \"kubernetes.io/projected/ec262e14-b98c-42c5-b319-1d988833a4b9-kube-api-access-n8stb\") pod \"ec262e14-b98c-42c5-b319-1d988833a4b9\" (UID: \"ec262e14-b98c-42c5-b319-1d988833a4b9\") "
Sep 17 18:08:04.880811 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.880510    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec262e14-b98c-42c5-b319-1d988833a4b9-cabundle-cert\") pod \"ec262e14-b98c-42c5-b319-1d988833a4b9\" (UID: \"ec262e14-b98c-42c5-b319-1d988833a4b9\") "
Sep 17 18:08:04.880811 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.880686    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec262e14-b98c-42c5-b319-1d988833a4b9-model-cache" (OuterVolumeSpecName: "model-cache") pod "ec262e14-b98c-42c5-b319-1d988833a4b9" (UID: "ec262e14-b98c-42c5-b319-1d988833a4b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:08:04.880811 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.880697    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec262e14-b98c-42c5-b319-1d988833a4b9-home" (OuterVolumeSpecName: "home") pod "ec262e14-b98c-42c5-b319-1d988833a4b9" (UID: "ec262e14-b98c-42c5-b319-1d988833a4b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:08:04.880811 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.880711    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec262e14-b98c-42c5-b319-1d988833a4b9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ec262e14-b98c-42c5-b319-1d988833a4b9" (UID: "ec262e14-b98c-42c5-b319-1d988833a4b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:08:04.881045 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.881015    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec262e14-b98c-42c5-b319-1d988833a4b9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ec262e14-b98c-42c5-b319-1d988833a4b9" (UID: "ec262e14-b98c-42c5-b319-1d988833a4b9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:08:04.882795 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.882761    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec262e14-b98c-42c5-b319-1d988833a4b9-dshm" (OuterVolumeSpecName: "dshm") pod "ec262e14-b98c-42c5-b319-1d988833a4b9" (UID: "ec262e14-b98c-42c5-b319-1d988833a4b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:08:04.882911 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.882874    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec262e14-b98c-42c5-b319-1d988833a4b9-kube-api-access-n8stb" (OuterVolumeSpecName: "kube-api-access-n8stb") pod "ec262e14-b98c-42c5-b319-1d988833a4b9" (UID: "ec262e14-b98c-42c5-b319-1d988833a4b9"). InnerVolumeSpecName "kube-api-access-n8stb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:08:04.883062 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.883034    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec262e14-b98c-42c5-b319-1d988833a4b9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ec262e14-b98c-42c5-b319-1d988833a4b9" (UID: "ec262e14-b98c-42c5-b319-1d988833a4b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:08:04.912388 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.912326    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec262e14-b98c-42c5-b319-1d988833a4b9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ec262e14-b98c-42c5-b319-1d988833a4b9" (UID: "ec262e14-b98c-42c5-b319-1d988833a4b9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:08:04.981807 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.981773    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec262e14-b98c-42c5-b319-1d988833a4b9-model-cache\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:08:04.981916 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.981810    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec262e14-b98c-42c5-b319-1d988833a4b9-kserve-provision-location\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:08:04.981916 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.981829    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec262e14-b98c-42c5-b319-1d988833a4b9-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:08:04.981916 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.981846    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8stb\" (UniqueName: \"kubernetes.io/projected/ec262e14-b98c-42c5-b319-1d988833a4b9-kube-api-access-n8stb\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:08:04.981916 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.981859    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec262e14-b98c-42c5-b319-1d988833a4b9-cabundle-cert\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:08:04.981916 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.981872    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec262e14-b98c-42c5-b319-1d988833a4b9-dshm\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:08:04.981916 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.981884    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec262e14-b98c-42c5-b319-1d988833a4b9-home\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:08:04.981916 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:04.981896    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec262e14-b98c-42c5-b319-1d988833a4b9-tmp-dir\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:08:05.097872 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:05.097836    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"]
Sep 17 18:08:05.101921 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:05.101895    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr"]
Sep 17 18:08:05.664898 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:05.664857    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-85f864cf88-pr5gr" podUID="ec262e14-b98c-42c5-b319-1d988833a4b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": context deadline exceeded"
Sep 17 18:08:07.040371 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:07.040341    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec262e14-b98c-42c5-b319-1d988833a4b9" path="/var/lib/kubelet/pods/ec262e14-b98c-42c5-b319-1d988833a4b9/volumes"
Sep 17 18:08:12.426481 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:12.426436    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4" podUID="7a7152b8-5386-45b2-b110-7dea7bd9ba4f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 17 18:08:22.427286 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:22.427235    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4" podUID="7a7152b8-5386-45b2-b110-7dea7bd9ba4f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 17 18:08:32.426925 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:32.426831    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4" podUID="7a7152b8-5386-45b2-b110-7dea7bd9ba4f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 17 18:08:42.436435 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:42.436398    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"
Sep 17 18:08:42.444185 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:42.444158    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"
Sep 17 18:08:44.061896 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:44.061866    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"]
Sep 17 18:08:59.084435 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.084378    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4" podUID="7a7152b8-5386-45b2-b110-7dea7bd9ba4f" containerName="main" containerID="cri-o://b0b448b9a74f3280eb58f5eadd6f4f168be7d4ab6e504cff95583887e451608d" gracePeriod=45
Sep 17 18:08:59.437938 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.437888    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4" podUID="7a7152b8-5386-45b2-b110-7dea7bd9ba4f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Sep 17 18:08:59.630587 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.630557    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"]
Sep 17 18:08:59.630893 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.630878    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec262e14-b98c-42c5-b319-1d988833a4b9" containerName="main"
Sep 17 18:08:59.630939 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.630897    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec262e14-b98c-42c5-b319-1d988833a4b9" containerName="main"
Sep 17 18:08:59.630939 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.630918    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec262e14-b98c-42c5-b319-1d988833a4b9" containerName="storage-initializer"
Sep 17 18:08:59.630939 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.630923    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec262e14-b98c-42c5-b319-1d988833a4b9" containerName="storage-initializer"
Sep 17 18:08:59.631034 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.631003    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec262e14-b98c-42c5-b319-1d988833a4b9" containerName="main"
Sep 17 18:08:59.632945 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.632924    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"
Sep 17 18:08:59.636189 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.636162    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 17 18:08:59.636189 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.636175    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 17 18:08:59.636360 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.636175    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:08:59.637145 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.637121    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 17 18:08:59.637247 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.637168    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-dzp2d\""
Sep 17 18:08:59.637247 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.637183    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 17 18:08:59.648733 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.648709    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"]
Sep 17 18:08:59.703001 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.702975    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-dshm\") pod \"tokenizer-template-args-test-kserve-79dbd77788-p2pgr\" (UID: \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"
Sep 17 18:08:59.703107 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.703007    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-model-cache\") pod \"tokenizer-template-args-test-kserve-79dbd77788-p2pgr\" (UID: \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"
Sep 17 18:08:59.703107 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.703041    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-79dbd77788-p2pgr\" (UID: \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"
Sep 17 18:08:59.703107 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.703070    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-79dbd77788-p2pgr\" (UID: \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"
Sep 17 18:08:59.703107 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.703096    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm5cg\" (UniqueName: \"kubernetes.io/projected/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-kube-api-access-jm5cg\") pod \"tokenizer-template-args-test-kserve-79dbd77788-p2pgr\" (UID: \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"
Sep 17 18:08:59.703241 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.703120    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-home\") pod \"tokenizer-template-args-test-kserve-79dbd77788-p2pgr\" (UID: \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"
Sep 17 18:08:59.703241 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.703141    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-tls-certs\") pod \"tokenizer-template-args-test-kserve-79dbd77788-p2pgr\" (UID: \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"
Sep 17 18:08:59.703241 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.703182    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-tmp-dir\") pod \"tokenizer-template-args-test-kserve-79dbd77788-p2pgr\" (UID: \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"
Sep 17 18:08:59.804272 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.804227    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-dshm\") pod \"tokenizer-template-args-test-kserve-79dbd77788-p2pgr\" (UID: \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"
Sep 17 18:08:59.804428 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.804277    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-model-cache\") pod \"tokenizer-template-args-test-kserve-79dbd77788-p2pgr\" (UID: \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"
Sep 17 18:08:59.804428 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.804315    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-79dbd77788-p2pgr\" (UID: \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"
Sep 17 18:08:59.804428 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.804345    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-79dbd77788-p2pgr\" (UID: \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"
Sep 17 18:08:59.804428 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.804375    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm5cg\" (UniqueName: \"kubernetes.io/projected/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-kube-api-access-jm5cg\") pod \"tokenizer-template-args-test-kserve-79dbd77788-p2pgr\" (UID: \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"
Sep 17 18:08:59.804636 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.804429    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-home\") pod \"tokenizer-template-args-test-kserve-79dbd77788-p2pgr\" (UID: \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"
Sep 17 18:08:59.804636 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.804463    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-tls-certs\") pod \"tokenizer-template-args-test-kserve-79dbd77788-p2pgr\" (UID: \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"
Sep 17 18:08:59.804636 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.804525    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-tmp-dir\") pod \"tokenizer-template-args-test-kserve-79dbd77788-p2pgr\" (UID: \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"
Sep 17 18:08:59.804811 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.804778    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-model-cache\") pod \"tokenizer-template-args-test-kserve-79dbd77788-p2pgr\" (UID: \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"
Sep 17 18:08:59.804869 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.804806    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-79dbd77788-p2pgr\" (UID: \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"
Sep 17 18:08:59.804975 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.804954    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-tmp-dir\") pod \"tokenizer-template-args-test-kserve-79dbd77788-p2pgr\" (UID: \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"
Sep 17 18:08:59.805158 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.805138    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-home\") pod \"tokenizer-template-args-test-kserve-79dbd77788-p2pgr\" (UID: \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"
Sep 17 18:08:59.805339 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.805319    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-79dbd77788-p2pgr\" (UID: \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"
Sep 17 18:08:59.806720 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.806704    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-dshm\") pod \"tokenizer-template-args-test-kserve-79dbd77788-p2pgr\" (UID: \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"
Sep 17 18:08:59.807091 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.807070    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-tls-certs\") pod \"tokenizer-template-args-test-kserve-79dbd77788-p2pgr\" (UID: \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"
Sep 17 18:08:59.814322 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.814297    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm5cg\" (UniqueName: \"kubernetes.io/projected/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-kube-api-access-jm5cg\") pod \"tokenizer-template-args-test-kserve-79dbd77788-p2pgr\" (UID: \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"
Sep 17 18:08:59.828713 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.828696    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"
Sep 17 18:08:59.905756 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.905726    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-cabundle-cert\") pod \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\" (UID: \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\") "
Sep 17 18:08:59.905913 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.905770    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-home\") pod \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\" (UID: \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\") "
Sep 17 18:08:59.905913 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.905797    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-kserve-provision-location\") pod \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\" (UID: \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\") "
Sep 17 18:08:59.905913 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.905836    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jvbk\" (UniqueName: \"kubernetes.io/projected/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-kube-api-access-8jvbk\") pod \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\" (UID: \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\") "
Sep 17 18:08:59.905913 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.905869    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-dshm\") pod \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\" (UID: \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\") "
Sep 17 18:08:59.905913 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.905885    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-tmp-dir\") pod \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\" (UID: \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\") "
Sep 17 18:08:59.905913 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.905911    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-model-cache\") pod \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\" (UID: \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\") "
Sep 17 18:08:59.906228 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.905975    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-tls-certs\") pod \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\" (UID: \"7a7152b8-5386-45b2-b110-7dea7bd9ba4f\") "
Sep 17 18:08:59.906228 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.906176    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7a7152b8-5386-45b2-b110-7dea7bd9ba4f" (UID: "7a7152b8-5386-45b2-b110-7dea7bd9ba4f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:08:59.906548 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.906503    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-model-cache" (OuterVolumeSpecName: "model-cache") pod "7a7152b8-5386-45b2-b110-7dea7bd9ba4f" (UID: "7a7152b8-5386-45b2-b110-7dea7bd9ba4f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:08:59.906548 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.906524    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-home" (OuterVolumeSpecName: "home") pod "7a7152b8-5386-45b2-b110-7dea7bd9ba4f" (UID: "7a7152b8-5386-45b2-b110-7dea7bd9ba4f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:08:59.908408 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.908372    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7a7152b8-5386-45b2-b110-7dea7bd9ba4f" (UID: "7a7152b8-5386-45b2-b110-7dea7bd9ba4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:08:59.908514 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.908463    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-dshm" (OuterVolumeSpecName: "dshm") pod "7a7152b8-5386-45b2-b110-7dea7bd9ba4f" (UID: "7a7152b8-5386-45b2-b110-7dea7bd9ba4f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:08:59.908574 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.908544    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-kube-api-access-8jvbk" (OuterVolumeSpecName: "kube-api-access-8jvbk") pod "7a7152b8-5386-45b2-b110-7dea7bd9ba4f" (UID: "7a7152b8-5386-45b2-b110-7dea7bd9ba4f"). InnerVolumeSpecName "kube-api-access-8jvbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:08:59.919264 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.919235    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7a7152b8-5386-45b2-b110-7dea7bd9ba4f" (UID: "7a7152b8-5386-45b2-b110-7dea7bd9ba4f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:08:59.928731 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.928694    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7a7152b8-5386-45b2-b110-7dea7bd9ba4f" (UID: "7a7152b8-5386-45b2-b110-7dea7bd9ba4f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:08:59.958940 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:08:59.958867    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"
Sep 17 18:09:00.007013 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.006980    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jvbk\" (UniqueName: \"kubernetes.io/projected/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-kube-api-access-8jvbk\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:09:00.007013 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.007011    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-dshm\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:09:00.007255 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.007023    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-tmp-dir\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:09:00.007255 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.007038    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-model-cache\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:09:00.007255 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.007049    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:09:00.007255 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.007060    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-cabundle-cert\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:09:00.007255 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.007069    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-home\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:09:00.007255 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.007081    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a7152b8-5386-45b2-b110-7dea7bd9ba4f-kserve-provision-location\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:09:00.014173 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.014141    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd"]
Sep 17 18:09:00.014467 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.014454    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a7152b8-5386-45b2-b110-7dea7bd9ba4f" containerName="main"
Sep 17 18:09:00.014509 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.014469    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a7152b8-5386-45b2-b110-7dea7bd9ba4f" containerName="main"
Sep 17 18:09:00.014509 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.014478    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a7152b8-5386-45b2-b110-7dea7bd9ba4f" containerName="storage-initializer"
Sep 17 18:09:00.014509 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.014484    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a7152b8-5386-45b2-b110-7dea7bd9ba4f" containerName="storage-initializer"
Sep 17 18:09:00.014641 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.014538    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a7152b8-5386-45b2-b110-7dea7bd9ba4f" containerName="main"
Sep 17 18:09:00.017505 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.017477    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd"
Sep 17 18:09:00.021619 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.021428    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-epp-sa-dockercfg-tb8mr\""
Sep 17 18:09:00.025599 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.025570    2583 generic.go:356] "Generic (PLEG): container finished" podID="7a7152b8-5386-45b2-b110-7dea7bd9ba4f" containerID="b0b448b9a74f3280eb58f5eadd6f4f168be7d4ab6e504cff95583887e451608d" exitCode=0
Sep 17 18:09:00.025767 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.025739    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4" event={"ID":"7a7152b8-5386-45b2-b110-7dea7bd9ba4f","Type":"ContainerDied","Data":"b0b448b9a74f3280eb58f5eadd6f4f168be7d4ab6e504cff95583887e451608d"}
Sep 17 18:09:00.025877 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.025802    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4" event={"ID":"7a7152b8-5386-45b2-b110-7dea7bd9ba4f","Type":"ContainerDied","Data":"cc5d9b01574fba966950497c8547d2baa1fd540e1fd562f6b399df7b8fb2c86d"}
Sep 17 18:09:00.025877 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.025837    2583 scope.go:117] "RemoveContainer" containerID="b0b448b9a74f3280eb58f5eadd6f4f168be7d4ab6e504cff95583887e451608d"
Sep 17 18:09:00.025969 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.025889    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"
Sep 17 18:09:00.030265 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.030243    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd"]
Sep 17 18:09:00.038974 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.038949    2583 scope.go:117] "RemoveContainer" containerID="380fc88454acfc32ba8d17244418d8066e47959fbfc8c1ac12174382f5c81f0b"
Sep 17 18:09:00.048414 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.048395    2583 scope.go:117] "RemoveContainer" containerID="b0b448b9a74f3280eb58f5eadd6f4f168be7d4ab6e504cff95583887e451608d"
Sep 17 18:09:00.048683 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:09:00.048651    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0b448b9a74f3280eb58f5eadd6f4f168be7d4ab6e504cff95583887e451608d\": container with ID starting with b0b448b9a74f3280eb58f5eadd6f4f168be7d4ab6e504cff95583887e451608d not found: ID does not exist" containerID="b0b448b9a74f3280eb58f5eadd6f4f168be7d4ab6e504cff95583887e451608d"
Sep 17 18:09:00.048761 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.048689    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0b448b9a74f3280eb58f5eadd6f4f168be7d4ab6e504cff95583887e451608d"} err="failed to get container status \"b0b448b9a74f3280eb58f5eadd6f4f168be7d4ab6e504cff95583887e451608d\": rpc error: code = NotFound desc = could not find container \"b0b448b9a74f3280eb58f5eadd6f4f168be7d4ab6e504cff95583887e451608d\": container with ID starting with b0b448b9a74f3280eb58f5eadd6f4f168be7d4ab6e504cff95583887e451608d not found: ID does not exist"
Sep 17 18:09:00.048761 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.048708    2583 scope.go:117] "RemoveContainer" containerID="380fc88454acfc32ba8d17244418d8066e47959fbfc8c1ac12174382f5c81f0b"
Sep 17 18:09:00.048993 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:09:00.048977    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"380fc88454acfc32ba8d17244418d8066e47959fbfc8c1ac12174382f5c81f0b\": container with ID starting with 380fc88454acfc32ba8d17244418d8066e47959fbfc8c1ac12174382f5c81f0b not found: ID does not exist" containerID="380fc88454acfc32ba8d17244418d8066e47959fbfc8c1ac12174382f5c81f0b"
Sep 17 18:09:00.049040 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.048997    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"380fc88454acfc32ba8d17244418d8066e47959fbfc8c1ac12174382f5c81f0b"} err="failed to get container status \"380fc88454acfc32ba8d17244418d8066e47959fbfc8c1ac12174382f5c81f0b\": rpc error: code = NotFound desc = could not find container \"380fc88454acfc32ba8d17244418d8066e47959fbfc8c1ac12174382f5c81f0b\": container with ID starting with 380fc88454acfc32ba8d17244418d8066e47959fbfc8c1ac12174382f5c81f0b not found: ID does not exist"
Sep 17 18:09:00.066729 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.066704    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"]
Sep 17 18:09:00.072484 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.072462    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-d78cb44d9-q9wm4"]
Sep 17 18:09:00.108417 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.108393    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/432c1312-d3ce-4b5c-a395-d157e414e4ae-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd\" (UID: \"432c1312-d3ce-4b5c-a395-d157e414e4ae\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd"
Sep 17 18:09:00.108826 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.108421    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngpfz\" (UniqueName: \"kubernetes.io/projected/432c1312-d3ce-4b5c-a395-d157e414e4ae-kube-api-access-ngpfz\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd\" (UID: \"432c1312-d3ce-4b5c-a395-d157e414e4ae\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd"
Sep 17 18:09:00.209956 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.209877    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/432c1312-d3ce-4b5c-a395-d157e414e4ae-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd\" (UID: \"432c1312-d3ce-4b5c-a395-d157e414e4ae\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd"
Sep 17 18:09:00.209956 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.209915    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngpfz\" (UniqueName: \"kubernetes.io/projected/432c1312-d3ce-4b5c-a395-d157e414e4ae-kube-api-access-ngpfz\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd\" (UID: \"432c1312-d3ce-4b5c-a395-d157e414e4ae\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd"
Sep 17 18:09:00.212598 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.212570    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/432c1312-d3ce-4b5c-a395-d157e414e4ae-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd\" (UID: \"432c1312-d3ce-4b5c-a395-d157e414e4ae\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd"
Sep 17 18:09:00.222424 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.222401    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngpfz\" (UniqueName: \"kubernetes.io/projected/432c1312-d3ce-4b5c-a395-d157e414e4ae-kube-api-access-ngpfz\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd\" (UID: \"432c1312-d3ce-4b5c-a395-d157e414e4ae\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd"
Sep 17 18:09:00.244816 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.244795    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"]
Sep 17 18:09:00.247005 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:09:00.246978    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb9cbc3_b0c5_4f50_86ee_a1068fdf2370.slice/crio-bf49ec2ea266ef51ed9e084266dd0463ce9b719b1e130edd7eba7eb0ddeb58f5 WatchSource:0}: Error finding container bf49ec2ea266ef51ed9e084266dd0463ce9b719b1e130edd7eba7eb0ddeb58f5: Status 404 returned error can't find the container with id bf49ec2ea266ef51ed9e084266dd0463ce9b719b1e130edd7eba7eb0ddeb58f5
Sep 17 18:09:00.333644 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.333610    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd"
Sep 17 18:09:00.605633 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:00.605602    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd"]
Sep 17 18:09:00.608878 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:09:00.608847    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod432c1312_d3ce_4b5c_a395_d157e414e4ae.slice/crio-f0180636d60eb3dbc4a8565e5d066cd3a995af76df408d01d5ee608d861f3c76 WatchSource:0}: Error finding container f0180636d60eb3dbc4a8565e5d066cd3a995af76df408d01d5ee608d861f3c76: Status 404 returned error can't find the container with id f0180636d60eb3dbc4a8565e5d066cd3a995af76df408d01d5ee608d861f3c76
Sep 17 18:09:01.032312 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:01.032222    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd" event={"ID":"432c1312-d3ce-4b5c-a395-d157e414e4ae","Type":"ContainerStarted","Data":"84eb502dd56d83e98994e01e0c532178d4721ba4c4476234a4f57000b2ea5737"}
Sep 17 18:09:01.032312 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:01.032276    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd" event={"ID":"432c1312-d3ce-4b5c-a395-d157e414e4ae","Type":"ContainerStarted","Data":"f0180636d60eb3dbc4a8565e5d066cd3a995af76df408d01d5ee608d861f3c76"}
Sep 17 18:09:01.032618 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:01.032356    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd"
Sep 17 18:09:01.033908 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:01.033884    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr" event={"ID":"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370","Type":"ContainerStarted","Data":"47155ffaee2fdaf9fcda16230502f049e03d6a8879cbfba06f3955561c91d515"}
Sep 17 18:09:01.034031 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:01.033932    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr" event={"ID":"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370","Type":"ContainerStarted","Data":"bf49ec2ea266ef51ed9e084266dd0463ce9b719b1e130edd7eba7eb0ddeb58f5"}
Sep 17 18:09:01.039509 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:01.039486    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a7152b8-5386-45b2-b110-7dea7bd9ba4f" path="/var/lib/kubelet/pods/7a7152b8-5386-45b2-b110-7dea7bd9ba4f/volumes"
Sep 17 18:09:01.054850 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:01.054791    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd" podStartSLOduration=2.054772858 podStartE2EDuration="2.054772858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:09:01.053391129 +0000 UTC m=+2052.144697911" watchObservedRunningTime="2026-09-17 18:09:01.054772858 +0000 UTC m=+2052.146079624"
Sep 17 18:09:03.044346 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:03.044312    2583 generic.go:356] "Generic (PLEG): container finished" podID="efb9cbc3-b0c5-4f50-86ee-a1068fdf2370" containerID="47155ffaee2fdaf9fcda16230502f049e03d6a8879cbfba06f3955561c91d515" exitCode=0
Sep 17 18:09:03.044851 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:03.044384    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr" event={"ID":"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370","Type":"ContainerDied","Data":"47155ffaee2fdaf9fcda16230502f049e03d6a8879cbfba06f3955561c91d515"}
Sep 17 18:09:04.050781 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:04.050746    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr" event={"ID":"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370","Type":"ContainerStarted","Data":"448dcac6879db1838b3bbcd614fcfbfb1e54e10b6412a2670aee4b3ae048d6c5"}
Sep 17 18:09:04.072161 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:04.072116    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr" podStartSLOduration=2.927064986 podStartE2EDuration="5.072101528s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.145036542s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:09:04.071159275 +0000 UTC m=+2055.162466062" watchObservedRunningTime="2026-09-17 18:09:04.072101528 +0000 UTC m=+2055.163408296"
Sep 17 18:09:04.959351 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:04.959305    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"
Sep 17 18:09:09.959257 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:09.959223    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"
Sep 17 18:09:09.971526 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:09.971501    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"
Sep 17 18:09:10.085018 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:10.084983    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"
Sep 17 18:09:32.041751 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:32.041718    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd"
Sep 17 18:09:44.998454 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:44.998413    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"]
Sep 17 18:09:45.001740 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.001719    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"
Sep 17 18:09:45.004373 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.004349    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 17 18:09:45.004493 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.004377    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Sep 17 18:09:45.004493 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.004464    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 17 18:09:45.004638 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.004618    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-22zv4\""
Sep 17 18:09:45.004728 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.004687    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:09:45.004847 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.004827    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 17 18:09:45.013467 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.013447    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"]
Sep 17 18:09:45.074415 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.074386    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fda9c8f-4104-4716-b74b-d2ea25509456-home\") pod \"auth-disabled-test-kserve-77c457cc6b-b7gq7\" (UID: \"8fda9c8f-4104-4716-b74b-d2ea25509456\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"
Sep 17 18:09:45.074646 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.074422    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fda9c8f-4104-4716-b74b-d2ea25509456-model-cache\") pod \"auth-disabled-test-kserve-77c457cc6b-b7gq7\" (UID: \"8fda9c8f-4104-4716-b74b-d2ea25509456\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"
Sep 17 18:09:45.074646 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.074450    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fda9c8f-4104-4716-b74b-d2ea25509456-cabundle-cert\") pod \"auth-disabled-test-kserve-77c457cc6b-b7gq7\" (UID: \"8fda9c8f-4104-4716-b74b-d2ea25509456\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"
Sep 17 18:09:45.074646 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.074595    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6knbq\" (UniqueName: \"kubernetes.io/projected/8fda9c8f-4104-4716-b74b-d2ea25509456-kube-api-access-6knbq\") pod \"auth-disabled-test-kserve-77c457cc6b-b7gq7\" (UID: \"8fda9c8f-4104-4716-b74b-d2ea25509456\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"
Sep 17 18:09:45.074646 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.074641    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fda9c8f-4104-4716-b74b-d2ea25509456-tmp-dir\") pod \"auth-disabled-test-kserve-77c457cc6b-b7gq7\" (UID: \"8fda9c8f-4104-4716-b74b-d2ea25509456\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"
Sep 17 18:09:45.074866 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.074696    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fda9c8f-4104-4716-b74b-d2ea25509456-kserve-provision-location\") pod \"auth-disabled-test-kserve-77c457cc6b-b7gq7\" (UID: \"8fda9c8f-4104-4716-b74b-d2ea25509456\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"
Sep 17 18:09:45.074866 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.074729    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fda9c8f-4104-4716-b74b-d2ea25509456-dshm\") pod \"auth-disabled-test-kserve-77c457cc6b-b7gq7\" (UID: \"8fda9c8f-4104-4716-b74b-d2ea25509456\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"
Sep 17 18:09:45.074866 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.074761    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fda9c8f-4104-4716-b74b-d2ea25509456-tls-certs\") pod \"auth-disabled-test-kserve-77c457cc6b-b7gq7\" (UID: \"8fda9c8f-4104-4716-b74b-d2ea25509456\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"
Sep 17 18:09:45.175669 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.175624    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6knbq\" (UniqueName: \"kubernetes.io/projected/8fda9c8f-4104-4716-b74b-d2ea25509456-kube-api-access-6knbq\") pod \"auth-disabled-test-kserve-77c457cc6b-b7gq7\" (UID: \"8fda9c8f-4104-4716-b74b-d2ea25509456\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"
Sep 17 18:09:45.175833 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.175693    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fda9c8f-4104-4716-b74b-d2ea25509456-tmp-dir\") pod \"auth-disabled-test-kserve-77c457cc6b-b7gq7\" (UID: \"8fda9c8f-4104-4716-b74b-d2ea25509456\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"
Sep 17 18:09:45.175833 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.175715    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fda9c8f-4104-4716-b74b-d2ea25509456-kserve-provision-location\") pod \"auth-disabled-test-kserve-77c457cc6b-b7gq7\" (UID: \"8fda9c8f-4104-4716-b74b-d2ea25509456\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"
Sep 17 18:09:45.175833 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.175732    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fda9c8f-4104-4716-b74b-d2ea25509456-dshm\") pod \"auth-disabled-test-kserve-77c457cc6b-b7gq7\" (UID: \"8fda9c8f-4104-4716-b74b-d2ea25509456\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"
Sep 17 18:09:45.175833 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.175752    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fda9c8f-4104-4716-b74b-d2ea25509456-tls-certs\") pod \"auth-disabled-test-kserve-77c457cc6b-b7gq7\" (UID: \"8fda9c8f-4104-4716-b74b-d2ea25509456\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"
Sep 17 18:09:45.176028 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.175841    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fda9c8f-4104-4716-b74b-d2ea25509456-home\") pod \"auth-disabled-test-kserve-77c457cc6b-b7gq7\" (UID: \"8fda9c8f-4104-4716-b74b-d2ea25509456\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"
Sep 17 18:09:45.176028 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.175886    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fda9c8f-4104-4716-b74b-d2ea25509456-model-cache\") pod \"auth-disabled-test-kserve-77c457cc6b-b7gq7\" (UID: \"8fda9c8f-4104-4716-b74b-d2ea25509456\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"
Sep 17 18:09:45.176028 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.175928    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fda9c8f-4104-4716-b74b-d2ea25509456-cabundle-cert\") pod \"auth-disabled-test-kserve-77c457cc6b-b7gq7\" (UID: \"8fda9c8f-4104-4716-b74b-d2ea25509456\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"
Sep 17 18:09:45.176207 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.176182    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fda9c8f-4104-4716-b74b-d2ea25509456-kserve-provision-location\") pod \"auth-disabled-test-kserve-77c457cc6b-b7gq7\" (UID: \"8fda9c8f-4104-4716-b74b-d2ea25509456\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"
Sep 17 18:09:45.176294 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.176206    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fda9c8f-4104-4716-b74b-d2ea25509456-tmp-dir\") pod \"auth-disabled-test-kserve-77c457cc6b-b7gq7\" (UID: \"8fda9c8f-4104-4716-b74b-d2ea25509456\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"
Sep 17 18:09:45.176350 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.176290    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fda9c8f-4104-4716-b74b-d2ea25509456-home\") pod \"auth-disabled-test-kserve-77c457cc6b-b7gq7\" (UID: \"8fda9c8f-4104-4716-b74b-d2ea25509456\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"
Sep 17 18:09:45.176451 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.176432    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fda9c8f-4104-4716-b74b-d2ea25509456-model-cache\") pod \"auth-disabled-test-kserve-77c457cc6b-b7gq7\" (UID: \"8fda9c8f-4104-4716-b74b-d2ea25509456\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"
Sep 17 18:09:45.176700 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.176683    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fda9c8f-4104-4716-b74b-d2ea25509456-cabundle-cert\") pod \"auth-disabled-test-kserve-77c457cc6b-b7gq7\" (UID: \"8fda9c8f-4104-4716-b74b-d2ea25509456\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"
Sep 17 18:09:45.178105 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.178088    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fda9c8f-4104-4716-b74b-d2ea25509456-dshm\") pod \"auth-disabled-test-kserve-77c457cc6b-b7gq7\" (UID: \"8fda9c8f-4104-4716-b74b-d2ea25509456\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"
Sep 17 18:09:45.178456 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.178440    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fda9c8f-4104-4716-b74b-d2ea25509456-tls-certs\") pod \"auth-disabled-test-kserve-77c457cc6b-b7gq7\" (UID: \"8fda9c8f-4104-4716-b74b-d2ea25509456\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"
Sep 17 18:09:45.184379 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.184361    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6knbq\" (UniqueName: \"kubernetes.io/projected/8fda9c8f-4104-4716-b74b-d2ea25509456-kube-api-access-6knbq\") pod \"auth-disabled-test-kserve-77c457cc6b-b7gq7\" (UID: \"8fda9c8f-4104-4716-b74b-d2ea25509456\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"
Sep 17 18:09:45.313578 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.313499    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"
Sep 17 18:09:45.590497 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:45.590471    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"]
Sep 17 18:09:45.593119 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:09:45.593084    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fda9c8f_4104_4716_b74b_d2ea25509456.slice/crio-ef84b132feb294a0a18b356cd631f47e229ae6984e982a07c30b52a1698d79a6 WatchSource:0}: Error finding container ef84b132feb294a0a18b356cd631f47e229ae6984e982a07c30b52a1698d79a6: Status 404 returned error can't find the container with id ef84b132feb294a0a18b356cd631f47e229ae6984e982a07c30b52a1698d79a6
Sep 17 18:09:46.225301 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:46.225259    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7" event={"ID":"8fda9c8f-4104-4716-b74b-d2ea25509456","Type":"ContainerStarted","Data":"61d391137309eb2b9f4c7096bf540013c0bf9ee69d190468337a13ed2f2ca236"}
Sep 17 18:09:46.225685 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:46.225340    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7" event={"ID":"8fda9c8f-4104-4716-b74b-d2ea25509456","Type":"ContainerStarted","Data":"ef84b132feb294a0a18b356cd631f47e229ae6984e982a07c30b52a1698d79a6"}
Sep 17 18:09:48.236801 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:48.236769    2583 generic.go:356] "Generic (PLEG): container finished" podID="8fda9c8f-4104-4716-b74b-d2ea25509456" containerID="61d391137309eb2b9f4c7096bf540013c0bf9ee69d190468337a13ed2f2ca236" exitCode=0
Sep 17 18:09:48.237243 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:48.236853    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7" event={"ID":"8fda9c8f-4104-4716-b74b-d2ea25509456","Type":"ContainerDied","Data":"61d391137309eb2b9f4c7096bf540013c0bf9ee69d190468337a13ed2f2ca236"}
Sep 17 18:09:49.243710 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:49.243655    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7" event={"ID":"8fda9c8f-4104-4716-b74b-d2ea25509456","Type":"ContainerStarted","Data":"786295d3635294d67971e33dc351824a2d955fcc13b4f63d2b7666b830075497"}
Sep 17 18:09:49.268277 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:49.268219    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7" podStartSLOduration=3.15291377 podStartE2EDuration="5.268202544s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.115288774s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:09:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:09:49.266727864 +0000 UTC m=+2100.358034655" watchObservedRunningTime="2026-09-17 18:09:49.268202544 +0000 UTC m=+2100.359509309"
Sep 17 18:09:49.313646 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:49.313611    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"
Sep 17 18:09:55.313915 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:55.313874    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"
Sep 17 18:09:55.315542 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:09:55.315514    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7" podUID="8fda9c8f-4104-4716-b74b-d2ea25509456" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 17 18:10:05.314449 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:05.314400    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7" podUID="8fda9c8f-4104-4716-b74b-d2ea25509456" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 17 18:10:07.492400 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:07.492360    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"]
Sep 17 18:10:07.502014 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:07.501926    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd"]
Sep 17 18:10:07.503313 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:10:07.503276    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:10:07.503313 ip-10-0-141-30 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:10:07.503313 ip-10-0-141-30 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 18:10:07.503313 ip-10-0-141-30 kubenswrapper[2583]:  > containerID="448dcac6879db1838b3bbcd614fcfbfb1e54e10b6412a2670aee4b3ae048d6c5" cmd=["/bin/sleep","15"]
Sep 17 18:10:07.503674 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:07.503332    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr" message="" err=<
Sep 17 18:10:07.503674 ip-10-0-141-30 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:10:07.503674 ip-10-0-141-30 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 18:10:07.503674 ip-10-0-141-30 kubenswrapper[2583]:  >
Sep 17 18:10:07.503674 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:10:07.503351    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:10:07.503674 ip-10-0-141-30 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:10:07.503674 ip-10-0-141-30 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 18:10:07.503674 ip-10-0-141-30 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr" podUID="efb9cbc3-b0c5-4f50-86ee-a1068fdf2370" containerName="main" containerID="cri-o://448dcac6879db1838b3bbcd614fcfbfb1e54e10b6412a2670aee4b3ae048d6c5"
Sep 17 18:10:07.503674 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:07.503398    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr" podUID="efb9cbc3-b0c5-4f50-86ee-a1068fdf2370" containerName="main" containerID="cri-o://448dcac6879db1838b3bbcd614fcfbfb1e54e10b6412a2670aee4b3ae048d6c5" gracePeriod=60
Sep 17 18:10:07.854576 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:07.854545    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"
Sep 17 18:10:07.980894 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:07.980856    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-tmp-dir\") pod \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\" (UID: \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\") "
Sep 17 18:10:07.980894 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:07.980894    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-cabundle-cert\") pod \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\" (UID: \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\") "
Sep 17 18:10:07.981165 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:07.980933    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-home\") pod \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\" (UID: \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\") "
Sep 17 18:10:07.981165 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:07.980994    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-kserve-provision-location\") pod \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\" (UID: \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\") "
Sep 17 18:10:07.981165 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:07.981012    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-dshm\") pod \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\" (UID: \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\") "
Sep 17 18:10:07.981165 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:07.981033    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-tls-certs\") pod \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\" (UID: \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\") "
Sep 17 18:10:07.981165 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:07.981062    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jm5cg\" (UniqueName: \"kubernetes.io/projected/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-kube-api-access-jm5cg\") pod \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\" (UID: \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\") "
Sep 17 18:10:07.981165 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:07.981086    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-model-cache\") pod \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\" (UID: \"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370\") "
Sep 17 18:10:07.981165 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:07.981136    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "efb9cbc3-b0c5-4f50-86ee-a1068fdf2370" (UID: "efb9cbc3-b0c5-4f50-86ee-a1068fdf2370"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:10:07.981552 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:07.981209    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-home" (OuterVolumeSpecName: "home") pod "efb9cbc3-b0c5-4f50-86ee-a1068fdf2370" (UID: "efb9cbc3-b0c5-4f50-86ee-a1068fdf2370"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:10:07.981552 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:07.981328    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-model-cache" (OuterVolumeSpecName: "model-cache") pod "efb9cbc3-b0c5-4f50-86ee-a1068fdf2370" (UID: "efb9cbc3-b0c5-4f50-86ee-a1068fdf2370"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:10:07.981552 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:07.981342    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-tmp-dir\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:10:07.981552 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:07.981361    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-home\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:10:07.981552 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:07.981359    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "efb9cbc3-b0c5-4f50-86ee-a1068fdf2370" (UID: "efb9cbc3-b0c5-4f50-86ee-a1068fdf2370"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:10:07.983440 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:07.983409    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-kube-api-access-jm5cg" (OuterVolumeSpecName: "kube-api-access-jm5cg") pod "efb9cbc3-b0c5-4f50-86ee-a1068fdf2370" (UID: "efb9cbc3-b0c5-4f50-86ee-a1068fdf2370"). InnerVolumeSpecName "kube-api-access-jm5cg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:10:07.983567 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:07.983515    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "efb9cbc3-b0c5-4f50-86ee-a1068fdf2370" (UID: "efb9cbc3-b0c5-4f50-86ee-a1068fdf2370"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:10:07.983631 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:07.983599    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-dshm" (OuterVolumeSpecName: "dshm") pod "efb9cbc3-b0c5-4f50-86ee-a1068fdf2370" (UID: "efb9cbc3-b0c5-4f50-86ee-a1068fdf2370"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:10:08.003000 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:08.002973    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "efb9cbc3-b0c5-4f50-86ee-a1068fdf2370" (UID: "efb9cbc3-b0c5-4f50-86ee-a1068fdf2370"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:10:08.082514 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:08.082439    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm5cg\" (UniqueName: \"kubernetes.io/projected/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-kube-api-access-jm5cg\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:10:08.082514 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:08.082465    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-model-cache\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:10:08.082514 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:08.082474    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-cabundle-cert\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:10:08.082514 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:08.082481    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-kserve-provision-location\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:10:08.082514 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:08.082493    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-dshm\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:10:08.082514 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:08.082501    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:10:08.331907 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:08.331874    2583 generic.go:356] "Generic (PLEG): container finished" podID="efb9cbc3-b0c5-4f50-86ee-a1068fdf2370" containerID="448dcac6879db1838b3bbcd614fcfbfb1e54e10b6412a2670aee4b3ae048d6c5" exitCode=0
Sep 17 18:10:08.332087 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:08.331937    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"
Sep 17 18:10:08.332087 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:08.331948    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr" event={"ID":"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370","Type":"ContainerDied","Data":"448dcac6879db1838b3bbcd614fcfbfb1e54e10b6412a2670aee4b3ae048d6c5"}
Sep 17 18:10:08.332087 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:08.332001    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr" event={"ID":"efb9cbc3-b0c5-4f50-86ee-a1068fdf2370","Type":"ContainerDied","Data":"bf49ec2ea266ef51ed9e084266dd0463ce9b719b1e130edd7eba7eb0ddeb58f5"}
Sep 17 18:10:08.332087 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:08.332032    2583 scope.go:117] "RemoveContainer" containerID="448dcac6879db1838b3bbcd614fcfbfb1e54e10b6412a2670aee4b3ae048d6c5"
Sep 17 18:10:08.343257 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:08.343236    2583 scope.go:117] "RemoveContainer" containerID="47155ffaee2fdaf9fcda16230502f049e03d6a8879cbfba06f3955561c91d515"
Sep 17 18:10:08.350807 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:08.350790    2583 scope.go:117] "RemoveContainer" containerID="448dcac6879db1838b3bbcd614fcfbfb1e54e10b6412a2670aee4b3ae048d6c5"
Sep 17 18:10:08.351103 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:10:08.351085    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"448dcac6879db1838b3bbcd614fcfbfb1e54e10b6412a2670aee4b3ae048d6c5\": container with ID starting with 448dcac6879db1838b3bbcd614fcfbfb1e54e10b6412a2670aee4b3ae048d6c5 not found: ID does not exist" containerID="448dcac6879db1838b3bbcd614fcfbfb1e54e10b6412a2670aee4b3ae048d6c5"
Sep 17 18:10:08.351147 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:08.351110    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"448dcac6879db1838b3bbcd614fcfbfb1e54e10b6412a2670aee4b3ae048d6c5"} err="failed to get container status \"448dcac6879db1838b3bbcd614fcfbfb1e54e10b6412a2670aee4b3ae048d6c5\": rpc error: code = NotFound desc = could not find container \"448dcac6879db1838b3bbcd614fcfbfb1e54e10b6412a2670aee4b3ae048d6c5\": container with ID starting with 448dcac6879db1838b3bbcd614fcfbfb1e54e10b6412a2670aee4b3ae048d6c5 not found: ID does not exist"
Sep 17 18:10:08.351147 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:08.351124    2583 scope.go:117] "RemoveContainer" containerID="47155ffaee2fdaf9fcda16230502f049e03d6a8879cbfba06f3955561c91d515"
Sep 17 18:10:08.351353 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:10:08.351334    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47155ffaee2fdaf9fcda16230502f049e03d6a8879cbfba06f3955561c91d515\": container with ID starting with 47155ffaee2fdaf9fcda16230502f049e03d6a8879cbfba06f3955561c91d515 not found: ID does not exist" containerID="47155ffaee2fdaf9fcda16230502f049e03d6a8879cbfba06f3955561c91d515"
Sep 17 18:10:08.351415 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:08.351359    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47155ffaee2fdaf9fcda16230502f049e03d6a8879cbfba06f3955561c91d515"} err="failed to get container status \"47155ffaee2fdaf9fcda16230502f049e03d6a8879cbfba06f3955561c91d515\": rpc error: code = NotFound desc = could not find container \"47155ffaee2fdaf9fcda16230502f049e03d6a8879cbfba06f3955561c91d515\": container with ID starting with 47155ffaee2fdaf9fcda16230502f049e03d6a8879cbfba06f3955561c91d515 not found: ID does not exist"
Sep 17 18:10:08.354237 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:08.354216    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"]
Sep 17 18:10:08.358734 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:08.358712    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-79dbd77788-p2pgr"]
Sep 17 18:10:09.040640 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:09.040604    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efb9cbc3-b0c5-4f50-86ee-a1068fdf2370" path="/var/lib/kubelet/pods/efb9cbc3-b0c5-4f50-86ee-a1068fdf2370/volumes"
Sep 17 18:10:12.042393 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:12.042355    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd" podUID="432c1312-d3ce-4b5c-a395-d157e414e4ae" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:10:15.314213 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:15.314170    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7" podUID="8fda9c8f-4104-4716-b74b-d2ea25509456" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 17 18:10:22.042206 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:22.042157    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd" podUID="432c1312-d3ce-4b5c-a395-d157e414e4ae" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:10:22.522641 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:22.522569    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd" podUID="432c1312-d3ce-4b5c-a395-d157e414e4ae" containerName="main" containerID="cri-o://84eb502dd56d83e98994e01e0c532178d4721ba4c4476234a4f57000b2ea5737" gracePeriod=45
Sep 17 18:10:25.314257 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:25.314215    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7" podUID="8fda9c8f-4104-4716-b74b-d2ea25509456" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 17 18:10:32.042003 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:32.041960    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd" podUID="432c1312-d3ce-4b5c-a395-d157e414e4ae" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:10:32.042489 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:32.042057    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd"
Sep 17 18:10:35.314028 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:35.313900    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7" podUID="8fda9c8f-4104-4716-b74b-d2ea25509456" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 17 18:10:42.042363 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:42.042321    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd" podUID="432c1312-d3ce-4b5c-a395-d157e414e4ae" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:10:45.314699 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:45.314637    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7" podUID="8fda9c8f-4104-4716-b74b-d2ea25509456" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 17 18:10:52.042492 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:52.042449    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd" podUID="432c1312-d3ce-4b5c-a395-d157e414e4ae" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:10:52.769089 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:52.769065    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd"
Sep 17 18:10:52.843828 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:52.843751    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngpfz\" (UniqueName: \"kubernetes.io/projected/432c1312-d3ce-4b5c-a395-d157e414e4ae-kube-api-access-ngpfz\") pod \"432c1312-d3ce-4b5c-a395-d157e414e4ae\" (UID: \"432c1312-d3ce-4b5c-a395-d157e414e4ae\") "
Sep 17 18:10:52.843828 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:52.843803    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/432c1312-d3ce-4b5c-a395-d157e414e4ae-tls-certs\") pod \"432c1312-d3ce-4b5c-a395-d157e414e4ae\" (UID: \"432c1312-d3ce-4b5c-a395-d157e414e4ae\") "
Sep 17 18:10:52.845810 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:52.845781    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432c1312-d3ce-4b5c-a395-d157e414e4ae-kube-api-access-ngpfz" (OuterVolumeSpecName: "kube-api-access-ngpfz") pod "432c1312-d3ce-4b5c-a395-d157e414e4ae" (UID: "432c1312-d3ce-4b5c-a395-d157e414e4ae"). InnerVolumeSpecName "kube-api-access-ngpfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:10:52.845937 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:52.845917    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/432c1312-d3ce-4b5c-a395-d157e414e4ae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "432c1312-d3ce-4b5c-a395-d157e414e4ae" (UID: "432c1312-d3ce-4b5c-a395-d157e414e4ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:10:52.944972 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:52.944926    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngpfz\" (UniqueName: \"kubernetes.io/projected/432c1312-d3ce-4b5c-a395-d157e414e4ae-kube-api-access-ngpfz\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:10:52.944972 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:52.944964    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/432c1312-d3ce-4b5c-a395-d157e414e4ae-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:10:53.530765 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:53.530720    2583 generic.go:356] "Generic (PLEG): container finished" podID="432c1312-d3ce-4b5c-a395-d157e414e4ae" containerID="84eb502dd56d83e98994e01e0c532178d4721ba4c4476234a4f57000b2ea5737" exitCode=0
Sep 17 18:10:53.530765 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:53.530764    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd"
Sep 17 18:10:53.531264 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:53.530806    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd" event={"ID":"432c1312-d3ce-4b5c-a395-d157e414e4ae","Type":"ContainerDied","Data":"84eb502dd56d83e98994e01e0c532178d4721ba4c4476234a4f57000b2ea5737"}
Sep 17 18:10:53.531264 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:53.530864    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd" event={"ID":"432c1312-d3ce-4b5c-a395-d157e414e4ae","Type":"ContainerDied","Data":"f0180636d60eb3dbc4a8565e5d066cd3a995af76df408d01d5ee608d861f3c76"}
Sep 17 18:10:53.531264 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:53.530892    2583 scope.go:117] "RemoveContainer" containerID="84eb502dd56d83e98994e01e0c532178d4721ba4c4476234a4f57000b2ea5737"
Sep 17 18:10:53.538990 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:53.538969    2583 scope.go:117] "RemoveContainer" containerID="84eb502dd56d83e98994e01e0c532178d4721ba4c4476234a4f57000b2ea5737"
Sep 17 18:10:53.539273 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:10:53.539247    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84eb502dd56d83e98994e01e0c532178d4721ba4c4476234a4f57000b2ea5737\": container with ID starting with 84eb502dd56d83e98994e01e0c532178d4721ba4c4476234a4f57000b2ea5737 not found: ID does not exist" containerID="84eb502dd56d83e98994e01e0c532178d4721ba4c4476234a4f57000b2ea5737"
Sep 17 18:10:53.539383 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:53.539276    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84eb502dd56d83e98994e01e0c532178d4721ba4c4476234a4f57000b2ea5737"} err="failed to get container status \"84eb502dd56d83e98994e01e0c532178d4721ba4c4476234a4f57000b2ea5737\": rpc error: code = NotFound desc = could not find container \"84eb502dd56d83e98994e01e0c532178d4721ba4c4476234a4f57000b2ea5737\": container with ID starting with 84eb502dd56d83e98994e01e0c532178d4721ba4c4476234a4f57000b2ea5737 not found: ID does not exist"
Sep 17 18:10:53.549837 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:53.549810    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd"]
Sep 17 18:10:53.552907 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:53.552883    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzprrd"]
Sep 17 18:10:55.040074 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:55.040040    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="432c1312-d3ce-4b5c-a395-d157e414e4ae" path="/var/lib/kubelet/pods/432c1312-d3ce-4b5c-a395-d157e414e4ae/volumes"
Sep 17 18:10:55.314649 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:10:55.314556    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7" podUID="8fda9c8f-4104-4716-b74b-d2ea25509456" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 17 18:11:05.314856 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:05.314813    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7" podUID="8fda9c8f-4104-4716-b74b-d2ea25509456" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 17 18:11:13.191711 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.191653    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5"]
Sep 17 18:11:13.192177 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.191980    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efb9cbc3-b0c5-4f50-86ee-a1068fdf2370" containerName="main"
Sep 17 18:11:13.192177 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.191991    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="efb9cbc3-b0c5-4f50-86ee-a1068fdf2370" containerName="main"
Sep 17 18:11:13.192177 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.192003    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="432c1312-d3ce-4b5c-a395-d157e414e4ae" containerName="main"
Sep 17 18:11:13.192177 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.192008    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="432c1312-d3ce-4b5c-a395-d157e414e4ae" containerName="main"
Sep 17 18:11:13.192177 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.192015    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efb9cbc3-b0c5-4f50-86ee-a1068fdf2370" containerName="storage-initializer"
Sep 17 18:11:13.192177 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.192022    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="efb9cbc3-b0c5-4f50-86ee-a1068fdf2370" containerName="storage-initializer"
Sep 17 18:11:13.192177 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.192068    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="432c1312-d3ce-4b5c-a395-d157e414e4ae" containerName="main"
Sep 17 18:11:13.192177 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.192076    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="efb9cbc3-b0c5-4f50-86ee-a1068fdf2370" containerName="main"
Sep 17 18:11:13.196291 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.196267    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5"
Sep 17 18:11:13.199450 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.199424    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 17 18:11:13.200406 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.200387    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 17 18:11:13.200499 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.200452    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-clmxc\""
Sep 17 18:11:13.200558 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.200527    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 17 18:11:13.213681 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.211178    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5"]
Sep 17 18:11:13.317149 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.317112    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8585945e-3896-4e4c-9c04-c5c39fde3006-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5\" (UID: \"8585945e-3896-4e4c-9c04-c5c39fde3006\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5"
Sep 17 18:11:13.317385 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.317173    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8585945e-3896-4e4c-9c04-c5c39fde3006-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5\" (UID: \"8585945e-3896-4e4c-9c04-c5c39fde3006\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5"
Sep 17 18:11:13.317385 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.317217    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8585945e-3896-4e4c-9c04-c5c39fde3006-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5\" (UID: \"8585945e-3896-4e4c-9c04-c5c39fde3006\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5"
Sep 17 18:11:13.317385 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.317238    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8585945e-3896-4e4c-9c04-c5c39fde3006-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5\" (UID: \"8585945e-3896-4e4c-9c04-c5c39fde3006\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5"
Sep 17 18:11:13.317385 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.317262    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzzgs\" (UniqueName: \"kubernetes.io/projected/8585945e-3896-4e4c-9c04-c5c39fde3006-kube-api-access-tzzgs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5\" (UID: \"8585945e-3896-4e4c-9c04-c5c39fde3006\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5"
Sep 17 18:11:13.317385 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.317293    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8585945e-3896-4e4c-9c04-c5c39fde3006-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5\" (UID: \"8585945e-3896-4e4c-9c04-c5c39fde3006\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5"
Sep 17 18:11:13.396405 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.396371    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb"]
Sep 17 18:11:13.399444 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.399419    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb"
Sep 17 18:11:13.401848 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.401828    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-kjq62\""
Sep 17 18:11:13.412838 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.412815    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb"]
Sep 17 18:11:13.418444 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.418419    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzzgs\" (UniqueName: \"kubernetes.io/projected/8585945e-3896-4e4c-9c04-c5c39fde3006-kube-api-access-tzzgs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5\" (UID: \"8585945e-3896-4e4c-9c04-c5c39fde3006\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5"
Sep 17 18:11:13.418559 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.418452    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8585945e-3896-4e4c-9c04-c5c39fde3006-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5\" (UID: \"8585945e-3896-4e4c-9c04-c5c39fde3006\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5"
Sep 17 18:11:13.418559 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.418489    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8585945e-3896-4e4c-9c04-c5c39fde3006-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5\" (UID: \"8585945e-3896-4e4c-9c04-c5c39fde3006\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5"
Sep 17 18:11:13.418559 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.418533    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8585945e-3896-4e4c-9c04-c5c39fde3006-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5\" (UID: \"8585945e-3896-4e4c-9c04-c5c39fde3006\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5"
Sep 17 18:11:13.418763 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.418570    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8585945e-3896-4e4c-9c04-c5c39fde3006-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5\" (UID: \"8585945e-3896-4e4c-9c04-c5c39fde3006\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5"
Sep 17 18:11:13.418763 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.418601    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8585945e-3896-4e4c-9c04-c5c39fde3006-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5\" (UID: \"8585945e-3896-4e4c-9c04-c5c39fde3006\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5"
Sep 17 18:11:13.418994 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.418962    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8585945e-3896-4e4c-9c04-c5c39fde3006-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5\" (UID: \"8585945e-3896-4e4c-9c04-c5c39fde3006\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5"
Sep 17 18:11:13.419106 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.419055    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8585945e-3896-4e4c-9c04-c5c39fde3006-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5\" (UID: \"8585945e-3896-4e4c-9c04-c5c39fde3006\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5"
Sep 17 18:11:13.419181 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.419102    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8585945e-3896-4e4c-9c04-c5c39fde3006-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5\" (UID: \"8585945e-3896-4e4c-9c04-c5c39fde3006\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5"
Sep 17 18:11:13.421249 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.421215    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8585945e-3896-4e4c-9c04-c5c39fde3006-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5\" (UID: \"8585945e-3896-4e4c-9c04-c5c39fde3006\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5"
Sep 17 18:11:13.421357 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.421219    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8585945e-3896-4e4c-9c04-c5c39fde3006-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5\" (UID: \"8585945e-3896-4e4c-9c04-c5c39fde3006\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5"
Sep 17 18:11:13.431822 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.431795    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzzgs\" (UniqueName: \"kubernetes.io/projected/8585945e-3896-4e4c-9c04-c5c39fde3006-kube-api-access-tzzgs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5\" (UID: \"8585945e-3896-4e4c-9c04-c5c39fde3006\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5"
Sep 17 18:11:13.506222 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.506143    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5"
Sep 17 18:11:13.519694 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.519632    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e34d56b-00fc-4d2c-85d8-2aec203764a0-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb\" (UID: \"2e34d56b-00fc-4d2c-85d8-2aec203764a0\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb"
Sep 17 18:11:13.519966 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.519928    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwww9\" (UniqueName: \"kubernetes.io/projected/2e34d56b-00fc-4d2c-85d8-2aec203764a0-kube-api-access-jwww9\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb\" (UID: \"2e34d56b-00fc-4d2c-85d8-2aec203764a0\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb"
Sep 17 18:11:13.620929 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.620897    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e34d56b-00fc-4d2c-85d8-2aec203764a0-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb\" (UID: \"2e34d56b-00fc-4d2c-85d8-2aec203764a0\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb"
Sep 17 18:11:13.621100 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.620967    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwww9\" (UniqueName: \"kubernetes.io/projected/2e34d56b-00fc-4d2c-85d8-2aec203764a0-kube-api-access-jwww9\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb\" (UID: \"2e34d56b-00fc-4d2c-85d8-2aec203764a0\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb"
Sep 17 18:11:13.623696 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.623651    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e34d56b-00fc-4d2c-85d8-2aec203764a0-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb\" (UID: \"2e34d56b-00fc-4d2c-85d8-2aec203764a0\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb"
Sep 17 18:11:13.629624 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.629598    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwww9\" (UniqueName: \"kubernetes.io/projected/2e34d56b-00fc-4d2c-85d8-2aec203764a0-kube-api-access-jwww9\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb\" (UID: \"2e34d56b-00fc-4d2c-85d8-2aec203764a0\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb"
Sep 17 18:11:13.709487 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.709454    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb"
Sep 17 18:11:13.786625 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.786590    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5"]
Sep 17 18:11:13.790176 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:11:13.790147    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8585945e_3896_4e4c_9c04_c5c39fde3006.slice/crio-3fe8d90f6a2ebfe91d93fe479ae59721a5d20a1bc14bb189aee4b5fb8d0bc844 WatchSource:0}: Error finding container 3fe8d90f6a2ebfe91d93fe479ae59721a5d20a1bc14bb189aee4b5fb8d0bc844: Status 404 returned error can't find the container with id 3fe8d90f6a2ebfe91d93fe479ae59721a5d20a1bc14bb189aee4b5fb8d0bc844
Sep 17 18:11:13.985383 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:13.985222    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb"]
Sep 17 18:11:13.987949 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:11:13.987922    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e34d56b_00fc_4d2c_85d8_2aec203764a0.slice/crio-bf9c294e278725773d16021533186efbbcc6d6df364ca347dc056bfdf3093c0e WatchSource:0}: Error finding container bf9c294e278725773d16021533186efbbcc6d6df364ca347dc056bfdf3093c0e: Status 404 returned error can't find the container with id bf9c294e278725773d16021533186efbbcc6d6df364ca347dc056bfdf3093c0e
Sep 17 18:11:14.634403 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:14.634361    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb" event={"ID":"2e34d56b-00fc-4d2c-85d8-2aec203764a0","Type":"ContainerStarted","Data":"c4efc1fad4c90e47460bf75ae7949b8ed1bfc5b826987a329518320c033b0529"}
Sep 17 18:11:14.634894 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:14.634428    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb"
Sep 17 18:11:14.634894 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:14.634455    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb" event={"ID":"2e34d56b-00fc-4d2c-85d8-2aec203764a0","Type":"ContainerStarted","Data":"bf9c294e278725773d16021533186efbbcc6d6df364ca347dc056bfdf3093c0e"}
Sep 17 18:11:14.635934 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:14.635909    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5" event={"ID":"8585945e-3896-4e4c-9c04-c5c39fde3006","Type":"ContainerStarted","Data":"99f0c364ecc8479bd3e0c07d88d3d91066164749206668479f854f5184bf1d28"}
Sep 17 18:11:14.636039 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:14.635946    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5" event={"ID":"8585945e-3896-4e4c-9c04-c5c39fde3006","Type":"ContainerStarted","Data":"3fe8d90f6a2ebfe91d93fe479ae59721a5d20a1bc14bb189aee4b5fb8d0bc844"}
Sep 17 18:11:14.654814 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:14.654756    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb" podStartSLOduration=1.654737303 podStartE2EDuration="1.654737303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:11:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:11:14.653957602 +0000 UTC m=+2185.745264425" watchObservedRunningTime="2026-09-17 18:11:14.654737303 +0000 UTC m=+2185.746044064"
Sep 17 18:11:14.678171 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:14.678116    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5" podStartSLOduration=1.678098183 podStartE2EDuration="1.678098183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:11:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:11:14.67632646 +0000 UTC m=+2185.767633270" watchObservedRunningTime="2026-09-17 18:11:14.678098183 +0000 UTC m=+2185.769404948"
Sep 17 18:11:15.314324 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:15.314282    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7" podUID="8fda9c8f-4104-4716-b74b-d2ea25509456" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 17 18:11:15.506421 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:15.506381    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5"
Sep 17 18:11:23.506768 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:23.506723    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5"
Sep 17 18:11:23.519184 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:23.519157    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5"
Sep 17 18:11:23.687869 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:23.687841    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5"
Sep 17 18:11:25.314055 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:25.314013    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7" podUID="8fda9c8f-4104-4716-b74b-d2ea25509456" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 17 18:11:35.314855 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:35.314807    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7" podUID="8fda9c8f-4104-4716-b74b-d2ea25509456" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 17 18:11:45.324527 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:45.324493    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"
Sep 17 18:11:45.333090 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:45.333061    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"
Sep 17 18:11:45.641275 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:45.641243    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb"
Sep 17 18:11:49.822797 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:49.822737    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb"]
Sep 17 18:11:49.824839 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:49.824800    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5"]
Sep 17 18:11:49.835202 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:11:49.835164    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:11:49.835202 ip-10-0-141-30 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:11:49.835202 ip-10-0-141-30 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 18:11:49.835202 ip-10-0-141-30 kubenswrapper[2583]:  > containerID="99f0c364ecc8479bd3e0c07d88d3d91066164749206668479f854f5184bf1d28" cmd=["/bin/sleep","15"]
Sep 17 18:11:49.835550 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:49.835217    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5" message="" err=<
Sep 17 18:11:49.835550 ip-10-0-141-30 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:11:49.835550 ip-10-0-141-30 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 18:11:49.835550 ip-10-0-141-30 kubenswrapper[2583]:  >
Sep 17 18:11:49.835550 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:11:49.835235    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:11:49.835550 ip-10-0-141-30 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:11:49.835550 ip-10-0-141-30 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 18:11:49.835550 ip-10-0-141-30 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5" podUID="8585945e-3896-4e4c-9c04-c5c39fde3006" containerName="main" containerID="cri-o://99f0c364ecc8479bd3e0c07d88d3d91066164749206668479f854f5184bf1d28"
Sep 17 18:11:49.835550 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:49.835272    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5" podUID="8585945e-3896-4e4c-9c04-c5c39fde3006" containerName="main" containerID="cri-o://99f0c364ecc8479bd3e0c07d88d3d91066164749206668479f854f5184bf1d28" gracePeriod=60
Sep 17 18:11:50.178682 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:50.178646    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5"
Sep 17 18:11:50.328075 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:50.328041    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8585945e-3896-4e4c-9c04-c5c39fde3006-tmp-dir\") pod \"8585945e-3896-4e4c-9c04-c5c39fde3006\" (UID: \"8585945e-3896-4e4c-9c04-c5c39fde3006\") "
Sep 17 18:11:50.328280 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:50.328128    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzzgs\" (UniqueName: \"kubernetes.io/projected/8585945e-3896-4e4c-9c04-c5c39fde3006-kube-api-access-tzzgs\") pod \"8585945e-3896-4e4c-9c04-c5c39fde3006\" (UID: \"8585945e-3896-4e4c-9c04-c5c39fde3006\") "
Sep 17 18:11:50.328280 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:50.328171    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8585945e-3896-4e4c-9c04-c5c39fde3006-tls-certs\") pod \"8585945e-3896-4e4c-9c04-c5c39fde3006\" (UID: \"8585945e-3896-4e4c-9c04-c5c39fde3006\") "
Sep 17 18:11:50.328280 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:50.328202    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8585945e-3896-4e4c-9c04-c5c39fde3006-model-cache\") pod \"8585945e-3896-4e4c-9c04-c5c39fde3006\" (UID: \"8585945e-3896-4e4c-9c04-c5c39fde3006\") "
Sep 17 18:11:50.328280 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:50.328233    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8585945e-3896-4e4c-9c04-c5c39fde3006-dshm\") pod \"8585945e-3896-4e4c-9c04-c5c39fde3006\" (UID: \"8585945e-3896-4e4c-9c04-c5c39fde3006\") "
Sep 17 18:11:50.328280 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:50.328259    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8585945e-3896-4e4c-9c04-c5c39fde3006-home\") pod \"8585945e-3896-4e4c-9c04-c5c39fde3006\" (UID: \"8585945e-3896-4e4c-9c04-c5c39fde3006\") "
Sep 17 18:11:50.328555 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:50.328520    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8585945e-3896-4e4c-9c04-c5c39fde3006-model-cache" (OuterVolumeSpecName: "model-cache") pod "8585945e-3896-4e4c-9c04-c5c39fde3006" (UID: "8585945e-3896-4e4c-9c04-c5c39fde3006"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:11:50.328649 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:50.328619    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8585945e-3896-4e4c-9c04-c5c39fde3006-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8585945e-3896-4e4c-9c04-c5c39fde3006" (UID: "8585945e-3896-4e4c-9c04-c5c39fde3006"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:11:50.328807 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:50.328648    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8585945e-3896-4e4c-9c04-c5c39fde3006-home" (OuterVolumeSpecName: "home") pod "8585945e-3896-4e4c-9c04-c5c39fde3006" (UID: "8585945e-3896-4e4c-9c04-c5c39fde3006"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:11:50.330386 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:50.330357    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8585945e-3896-4e4c-9c04-c5c39fde3006-dshm" (OuterVolumeSpecName: "dshm") pod "8585945e-3896-4e4c-9c04-c5c39fde3006" (UID: "8585945e-3896-4e4c-9c04-c5c39fde3006"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:11:50.330514 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:50.330494    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8585945e-3896-4e4c-9c04-c5c39fde3006-kube-api-access-tzzgs" (OuterVolumeSpecName: "kube-api-access-tzzgs") pod "8585945e-3896-4e4c-9c04-c5c39fde3006" (UID: "8585945e-3896-4e4c-9c04-c5c39fde3006"). InnerVolumeSpecName "kube-api-access-tzzgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:11:50.330706 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:50.330688    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8585945e-3896-4e4c-9c04-c5c39fde3006-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8585945e-3896-4e4c-9c04-c5c39fde3006" (UID: "8585945e-3896-4e4c-9c04-c5c39fde3006"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:11:50.428891 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:50.428815    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8585945e-3896-4e4c-9c04-c5c39fde3006-dshm\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:11:50.428891 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:50.428840    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8585945e-3896-4e4c-9c04-c5c39fde3006-home\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:11:50.428891 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:50.428867    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8585945e-3896-4e4c-9c04-c5c39fde3006-tmp-dir\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:11:50.428891 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:50.428877    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzzgs\" (UniqueName: \"kubernetes.io/projected/8585945e-3896-4e4c-9c04-c5c39fde3006-kube-api-access-tzzgs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:11:50.428891 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:50.428887    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8585945e-3896-4e4c-9c04-c5c39fde3006-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:11:50.428891 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:50.428896    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8585945e-3896-4e4c-9c04-c5c39fde3006-model-cache\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:11:50.803128 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:50.803029    2583 generic.go:356] "Generic (PLEG): container finished" podID="8585945e-3896-4e4c-9c04-c5c39fde3006" containerID="99f0c364ecc8479bd3e0c07d88d3d91066164749206668479f854f5184bf1d28" exitCode=0
Sep 17 18:11:50.803128 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:50.803080    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5"
Sep 17 18:11:50.803128 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:50.803113    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5" event={"ID":"8585945e-3896-4e4c-9c04-c5c39fde3006","Type":"ContainerDied","Data":"99f0c364ecc8479bd3e0c07d88d3d91066164749206668479f854f5184bf1d28"}
Sep 17 18:11:50.803348 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:50.803186    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5" event={"ID":"8585945e-3896-4e4c-9c04-c5c39fde3006","Type":"ContainerDied","Data":"3fe8d90f6a2ebfe91d93fe479ae59721a5d20a1bc14bb189aee4b5fb8d0bc844"}
Sep 17 18:11:50.803348 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:50.803207    2583 scope.go:117] "RemoveContainer" containerID="99f0c364ecc8479bd3e0c07d88d3d91066164749206668479f854f5184bf1d28"
Sep 17 18:11:50.812909 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:50.812891    2583 scope.go:117] "RemoveContainer" containerID="99f0c364ecc8479bd3e0c07d88d3d91066164749206668479f854f5184bf1d28"
Sep 17 18:11:50.813164 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:11:50.813147    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99f0c364ecc8479bd3e0c07d88d3d91066164749206668479f854f5184bf1d28\": container with ID starting with 99f0c364ecc8479bd3e0c07d88d3d91066164749206668479f854f5184bf1d28 not found: ID does not exist" containerID="99f0c364ecc8479bd3e0c07d88d3d91066164749206668479f854f5184bf1d28"
Sep 17 18:11:50.813219 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:50.813172    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99f0c364ecc8479bd3e0c07d88d3d91066164749206668479f854f5184bf1d28"} err="failed to get container status \"99f0c364ecc8479bd3e0c07d88d3d91066164749206668479f854f5184bf1d28\": rpc error: code = NotFound desc = could not find container \"99f0c364ecc8479bd3e0c07d88d3d91066164749206668479f854f5184bf1d28\": container with ID starting with 99f0c364ecc8479bd3e0c07d88d3d91066164749206668479f854f5184bf1d28 not found: ID does not exist"
Sep 17 18:11:50.837205 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:50.837178    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5"]
Sep 17 18:11:50.842760 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:50.842739    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-k6nr5"]
Sep 17 18:11:51.041588 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:51.041558    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8585945e-3896-4e4c-9c04-c5c39fde3006" path="/var/lib/kubelet/pods/8585945e-3896-4e4c-9c04-c5c39fde3006/volumes"
Sep 17 18:11:52.677066 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:52.677035    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"]
Sep 17 18:11:55.641839 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:11:55.641804    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb" podUID="2e34d56b-00fc-4d2c-85d8-2aec203764a0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:12:04.851566 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:04.851478    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb" podUID="2e34d56b-00fc-4d2c-85d8-2aec203764a0" containerName="main" containerID="cri-o://c4efc1fad4c90e47460bf75ae7949b8ed1bfc5b826987a329518320c033b0529" gracePeriod=45
Sep 17 18:12:05.641642 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:05.641602    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb" podUID="2e34d56b-00fc-4d2c-85d8-2aec203764a0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:12:07.699997 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:07.699922    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7" podUID="8fda9c8f-4104-4716-b74b-d2ea25509456" containerName="main" containerID="cri-o://786295d3635294d67971e33dc351824a2d955fcc13b4f63d2b7666b830075497" gracePeriod=45
Sep 17 18:12:08.325918 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.325878    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7" podUID="8fda9c8f-4104-4716-b74b-d2ea25509456" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 17 18:12:08.449460 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.449431    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"
Sep 17 18:12:08.578430 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.578351    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fda9c8f-4104-4716-b74b-d2ea25509456-kserve-provision-location\") pod \"8fda9c8f-4104-4716-b74b-d2ea25509456\" (UID: \"8fda9c8f-4104-4716-b74b-d2ea25509456\") "
Sep 17 18:12:08.578430 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.578396    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fda9c8f-4104-4716-b74b-d2ea25509456-model-cache\") pod \"8fda9c8f-4104-4716-b74b-d2ea25509456\" (UID: \"8fda9c8f-4104-4716-b74b-d2ea25509456\") "
Sep 17 18:12:08.578430 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.578420    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fda9c8f-4104-4716-b74b-d2ea25509456-tls-certs\") pod \"8fda9c8f-4104-4716-b74b-d2ea25509456\" (UID: \"8fda9c8f-4104-4716-b74b-d2ea25509456\") "
Sep 17 18:12:08.578741 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.578455    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fda9c8f-4104-4716-b74b-d2ea25509456-dshm\") pod \"8fda9c8f-4104-4716-b74b-d2ea25509456\" (UID: \"8fda9c8f-4104-4716-b74b-d2ea25509456\") "
Sep 17 18:12:08.578741 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.578484    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fda9c8f-4104-4716-b74b-d2ea25509456-tmp-dir\") pod \"8fda9c8f-4104-4716-b74b-d2ea25509456\" (UID: \"8fda9c8f-4104-4716-b74b-d2ea25509456\") "
Sep 17 18:12:08.578741 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.578519    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fda9c8f-4104-4716-b74b-d2ea25509456-home\") pod \"8fda9c8f-4104-4716-b74b-d2ea25509456\" (UID: \"8fda9c8f-4104-4716-b74b-d2ea25509456\") "
Sep 17 18:12:08.578741 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.578567    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fda9c8f-4104-4716-b74b-d2ea25509456-cabundle-cert\") pod \"8fda9c8f-4104-4716-b74b-d2ea25509456\" (UID: \"8fda9c8f-4104-4716-b74b-d2ea25509456\") "
Sep 17 18:12:08.578741 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.578623    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6knbq\" (UniqueName: \"kubernetes.io/projected/8fda9c8f-4104-4716-b74b-d2ea25509456-kube-api-access-6knbq\") pod \"8fda9c8f-4104-4716-b74b-d2ea25509456\" (UID: \"8fda9c8f-4104-4716-b74b-d2ea25509456\") "
Sep 17 18:12:08.578741 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.578701    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fda9c8f-4104-4716-b74b-d2ea25509456-model-cache" (OuterVolumeSpecName: "model-cache") pod "8fda9c8f-4104-4716-b74b-d2ea25509456" (UID: "8fda9c8f-4104-4716-b74b-d2ea25509456"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:12:08.579053 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.578978    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fda9c8f-4104-4716-b74b-d2ea25509456-model-cache\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:12:08.579143 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.579071    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fda9c8f-4104-4716-b74b-d2ea25509456-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8fda9c8f-4104-4716-b74b-d2ea25509456" (UID: "8fda9c8f-4104-4716-b74b-d2ea25509456"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:12:08.579279 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.579237    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fda9c8f-4104-4716-b74b-d2ea25509456-home" (OuterVolumeSpecName: "home") pod "8fda9c8f-4104-4716-b74b-d2ea25509456" (UID: "8fda9c8f-4104-4716-b74b-d2ea25509456"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:12:08.581308 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.581273    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fda9c8f-4104-4716-b74b-d2ea25509456-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8fda9c8f-4104-4716-b74b-d2ea25509456" (UID: "8fda9c8f-4104-4716-b74b-d2ea25509456"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:12:08.581407 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.581344    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fda9c8f-4104-4716-b74b-d2ea25509456-dshm" (OuterVolumeSpecName: "dshm") pod "8fda9c8f-4104-4716-b74b-d2ea25509456" (UID: "8fda9c8f-4104-4716-b74b-d2ea25509456"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:12:08.581407 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.581397    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fda9c8f-4104-4716-b74b-d2ea25509456-kube-api-access-6knbq" (OuterVolumeSpecName: "kube-api-access-6knbq") pod "8fda9c8f-4104-4716-b74b-d2ea25509456" (UID: "8fda9c8f-4104-4716-b74b-d2ea25509456"). InnerVolumeSpecName "kube-api-access-6knbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:12:08.596276 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.596250    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fda9c8f-4104-4716-b74b-d2ea25509456-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8fda9c8f-4104-4716-b74b-d2ea25509456" (UID: "8fda9c8f-4104-4716-b74b-d2ea25509456"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:12:08.600514 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.600493    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fda9c8f-4104-4716-b74b-d2ea25509456-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8fda9c8f-4104-4716-b74b-d2ea25509456" (UID: "8fda9c8f-4104-4716-b74b-d2ea25509456"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:12:08.679692 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.679643    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fda9c8f-4104-4716-b74b-d2ea25509456-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:12:08.679692 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.679689    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fda9c8f-4104-4716-b74b-d2ea25509456-dshm\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:12:08.679692 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.679697    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fda9c8f-4104-4716-b74b-d2ea25509456-tmp-dir\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:12:08.679905 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.679707    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fda9c8f-4104-4716-b74b-d2ea25509456-home\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:12:08.679905 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.679716    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8fda9c8f-4104-4716-b74b-d2ea25509456-cabundle-cert\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:12:08.679905 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.679724    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6knbq\" (UniqueName: \"kubernetes.io/projected/8fda9c8f-4104-4716-b74b-d2ea25509456-kube-api-access-6knbq\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:12:08.679905 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.679733    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fda9c8f-4104-4716-b74b-d2ea25509456-kserve-provision-location\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:12:08.883309 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.883273    2583 generic.go:356] "Generic (PLEG): container finished" podID="8fda9c8f-4104-4716-b74b-d2ea25509456" containerID="786295d3635294d67971e33dc351824a2d955fcc13b4f63d2b7666b830075497" exitCode=0
Sep 17 18:12:08.883743 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.883330    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"
Sep 17 18:12:08.883743 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.883354    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7" event={"ID":"8fda9c8f-4104-4716-b74b-d2ea25509456","Type":"ContainerDied","Data":"786295d3635294d67971e33dc351824a2d955fcc13b4f63d2b7666b830075497"}
Sep 17 18:12:08.883743 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.883416    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7" event={"ID":"8fda9c8f-4104-4716-b74b-d2ea25509456","Type":"ContainerDied","Data":"ef84b132feb294a0a18b356cd631f47e229ae6984e982a07c30b52a1698d79a6"}
Sep 17 18:12:08.883743 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.883439    2583 scope.go:117] "RemoveContainer" containerID="786295d3635294d67971e33dc351824a2d955fcc13b4f63d2b7666b830075497"
Sep 17 18:12:08.892149 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.892132    2583 scope.go:117] "RemoveContainer" containerID="61d391137309eb2b9f4c7096bf540013c0bf9ee69d190468337a13ed2f2ca236"
Sep 17 18:12:08.899741 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.899723    2583 scope.go:117] "RemoveContainer" containerID="786295d3635294d67971e33dc351824a2d955fcc13b4f63d2b7666b830075497"
Sep 17 18:12:08.900016 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:12:08.899998    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"786295d3635294d67971e33dc351824a2d955fcc13b4f63d2b7666b830075497\": container with ID starting with 786295d3635294d67971e33dc351824a2d955fcc13b4f63d2b7666b830075497 not found: ID does not exist" containerID="786295d3635294d67971e33dc351824a2d955fcc13b4f63d2b7666b830075497"
Sep 17 18:12:08.900080 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.900023    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"786295d3635294d67971e33dc351824a2d955fcc13b4f63d2b7666b830075497"} err="failed to get container status \"786295d3635294d67971e33dc351824a2d955fcc13b4f63d2b7666b830075497\": rpc error: code = NotFound desc = could not find container \"786295d3635294d67971e33dc351824a2d955fcc13b4f63d2b7666b830075497\": container with ID starting with 786295d3635294d67971e33dc351824a2d955fcc13b4f63d2b7666b830075497 not found: ID does not exist"
Sep 17 18:12:08.900080 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.900040    2583 scope.go:117] "RemoveContainer" containerID="61d391137309eb2b9f4c7096bf540013c0bf9ee69d190468337a13ed2f2ca236"
Sep 17 18:12:08.900286 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:12:08.900269    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61d391137309eb2b9f4c7096bf540013c0bf9ee69d190468337a13ed2f2ca236\": container with ID starting with 61d391137309eb2b9f4c7096bf540013c0bf9ee69d190468337a13ed2f2ca236 not found: ID does not exist" containerID="61d391137309eb2b9f4c7096bf540013c0bf9ee69d190468337a13ed2f2ca236"
Sep 17 18:12:08.900327 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.900292    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61d391137309eb2b9f4c7096bf540013c0bf9ee69d190468337a13ed2f2ca236"} err="failed to get container status \"61d391137309eb2b9f4c7096bf540013c0bf9ee69d190468337a13ed2f2ca236\": rpc error: code = NotFound desc = could not find container \"61d391137309eb2b9f4c7096bf540013c0bf9ee69d190468337a13ed2f2ca236\": container with ID starting with 61d391137309eb2b9f4c7096bf540013c0bf9ee69d190468337a13ed2f2ca236 not found: ID does not exist"
Sep 17 18:12:08.905443 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.905422    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"]
Sep 17 18:12:08.911405 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:08.911385    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-77c457cc6b-b7gq7"]
Sep 17 18:12:09.040452 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:09.040420    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fda9c8f-4104-4716-b74b-d2ea25509456" path="/var/lib/kubelet/pods/8fda9c8f-4104-4716-b74b-d2ea25509456/volumes"
Sep 17 18:12:15.641610 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:15.641573    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb" podUID="2e34d56b-00fc-4d2c-85d8-2aec203764a0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:12:15.642035 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:15.641693    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb"
Sep 17 18:12:25.641192 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:25.641149    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb" podUID="2e34d56b-00fc-4d2c-85d8-2aec203764a0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:12:34.992147 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:34.992107    2583 generic.go:356] "Generic (PLEG): container finished" podID="2e34d56b-00fc-4d2c-85d8-2aec203764a0" containerID="c4efc1fad4c90e47460bf75ae7949b8ed1bfc5b826987a329518320c033b0529" exitCode=0
Sep 17 18:12:34.992147 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:34.992149    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb" event={"ID":"2e34d56b-00fc-4d2c-85d8-2aec203764a0","Type":"ContainerDied","Data":"c4efc1fad4c90e47460bf75ae7949b8ed1bfc5b826987a329518320c033b0529"}
Sep 17 18:12:35.096803 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:35.096776    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb"
Sep 17 18:12:35.159162 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:35.159091    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwww9\" (UniqueName: \"kubernetes.io/projected/2e34d56b-00fc-4d2c-85d8-2aec203764a0-kube-api-access-jwww9\") pod \"2e34d56b-00fc-4d2c-85d8-2aec203764a0\" (UID: \"2e34d56b-00fc-4d2c-85d8-2aec203764a0\") "
Sep 17 18:12:35.159162 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:35.159124    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e34d56b-00fc-4d2c-85d8-2aec203764a0-tls-certs\") pod \"2e34d56b-00fc-4d2c-85d8-2aec203764a0\" (UID: \"2e34d56b-00fc-4d2c-85d8-2aec203764a0\") "
Sep 17 18:12:35.161161 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:35.161130    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e34d56b-00fc-4d2c-85d8-2aec203764a0-kube-api-access-jwww9" (OuterVolumeSpecName: "kube-api-access-jwww9") pod "2e34d56b-00fc-4d2c-85d8-2aec203764a0" (UID: "2e34d56b-00fc-4d2c-85d8-2aec203764a0"). InnerVolumeSpecName "kube-api-access-jwww9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:12:35.161262 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:35.161166    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e34d56b-00fc-4d2c-85d8-2aec203764a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e34d56b-00fc-4d2c-85d8-2aec203764a0" (UID: "2e34d56b-00fc-4d2c-85d8-2aec203764a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:12:35.260464 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:35.260432    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwww9\" (UniqueName: \"kubernetes.io/projected/2e34d56b-00fc-4d2c-85d8-2aec203764a0-kube-api-access-jwww9\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:12:35.260464 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:35.260457    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e34d56b-00fc-4d2c-85d8-2aec203764a0-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:12:35.998625 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:35.998587    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb" event={"ID":"2e34d56b-00fc-4d2c-85d8-2aec203764a0","Type":"ContainerDied","Data":"bf9c294e278725773d16021533186efbbcc6d6df364ca347dc056bfdf3093c0e"}
Sep 17 18:12:35.998625 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:35.998616    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb"
Sep 17 18:12:35.999179 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:35.998680    2583 scope.go:117] "RemoveContainer" containerID="c4efc1fad4c90e47460bf75ae7949b8ed1bfc5b826987a329518320c033b0529"
Sep 17 18:12:36.019381 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:36.019351    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb"]
Sep 17 18:12:36.027046 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:36.027025    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schepj5qb"]
Sep 17 18:12:37.039566 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:37.039530    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e34d56b-00fc-4d2c-85d8-2aec203764a0" path="/var/lib/kubelet/pods/2e34d56b-00fc-4d2c-85d8-2aec203764a0/volumes"
Sep 17 18:12:56.061770 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:56.061738    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4"]
Sep 17 18:12:56.062175 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:56.062017    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fda9c8f-4104-4716-b74b-d2ea25509456" containerName="main"
Sep 17 18:12:56.062175 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:56.062028    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fda9c8f-4104-4716-b74b-d2ea25509456" containerName="main"
Sep 17 18:12:56.062175 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:56.062035    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e34d56b-00fc-4d2c-85d8-2aec203764a0" containerName="main"
Sep 17 18:12:56.062175 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:56.062040    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e34d56b-00fc-4d2c-85d8-2aec203764a0" containerName="main"
Sep 17 18:12:56.062175 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:56.062054    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8585945e-3896-4e4c-9c04-c5c39fde3006" containerName="main"
Sep 17 18:12:56.062175 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:56.062060    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8585945e-3896-4e4c-9c04-c5c39fde3006" containerName="main"
Sep 17 18:12:56.062175 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:56.062071    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fda9c8f-4104-4716-b74b-d2ea25509456" containerName="storage-initializer"
Sep 17 18:12:56.062175 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:56.062076    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fda9c8f-4104-4716-b74b-d2ea25509456" containerName="storage-initializer"
Sep 17 18:12:56.062175 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:56.062114    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fda9c8f-4104-4716-b74b-d2ea25509456" containerName="main"
Sep 17 18:12:56.062175 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:56.062122    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8585945e-3896-4e4c-9c04-c5c39fde3006" containerName="main"
Sep 17 18:12:56.062175 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:56.062127    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e34d56b-00fc-4d2c-85d8-2aec203764a0" containerName="main"
Sep 17 18:12:56.066051 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:56.066027    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4"
Sep 17 18:12:56.068413 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:56.068390    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-6j5mz\""
Sep 17 18:12:56.069778 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:56.069751    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 17 18:12:56.069778 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:56.069772    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-m748j\""
Sep 17 18:12:56.069936 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:56.069856    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 17 18:12:56.069991 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:56.069942    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 17 18:12:56.073685 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:56.073646    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4"]
Sep 17 18:12:56.120371 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:56.120342    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffcjg\" (UniqueName: \"kubernetes.io/projected/05676388-253f-42af-addc-8d66dcf95259-kube-api-access-ffcjg\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4\" (UID: \"05676388-253f-42af-addc-8d66dcf95259\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4"
Sep 17 18:12:56.120511 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:56.120392    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05676388-253f-42af-addc-8d66dcf95259-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4\" (UID: \"05676388-253f-42af-addc-8d66dcf95259\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4"
Sep 17 18:12:56.221883 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:56.221836    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffcjg\" (UniqueName: \"kubernetes.io/projected/05676388-253f-42af-addc-8d66dcf95259-kube-api-access-ffcjg\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4\" (UID: \"05676388-253f-42af-addc-8d66dcf95259\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4"
Sep 17 18:12:56.222041 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:56.221898    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05676388-253f-42af-addc-8d66dcf95259-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4\" (UID: \"05676388-253f-42af-addc-8d66dcf95259\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4"
Sep 17 18:12:56.224447 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:56.224428    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05676388-253f-42af-addc-8d66dcf95259-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4\" (UID: \"05676388-253f-42af-addc-8d66dcf95259\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4"
Sep 17 18:12:56.230388 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:56.230360    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffcjg\" (UniqueName: \"kubernetes.io/projected/05676388-253f-42af-addc-8d66dcf95259-kube-api-access-ffcjg\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4\" (UID: \"05676388-253f-42af-addc-8d66dcf95259\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4"
Sep 17 18:12:56.377301 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:56.377275    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4"
Sep 17 18:12:56.648283 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:56.648253    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4"]
Sep 17 18:12:56.650702 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:12:56.650653    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05676388_253f_42af_addc_8d66dcf95259.slice/crio-3ae1e7b1029c8ed877db2cfdd82b5cec022c706c3d54d54c72202393c4aa4be4 WatchSource:0}: Error finding container 3ae1e7b1029c8ed877db2cfdd82b5cec022c706c3d54d54c72202393c4aa4be4: Status 404 returned error can't find the container with id 3ae1e7b1029c8ed877db2cfdd82b5cec022c706c3d54d54c72202393c4aa4be4
Sep 17 18:12:57.085086 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:57.084990    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4" event={"ID":"05676388-253f-42af-addc-8d66dcf95259","Type":"ContainerStarted","Data":"1d0ef7575189dada8e0e1a8890fbeed8e3d7b53280f6af9d2cad29990c89ad51"}
Sep 17 18:12:57.085086 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:57.085041    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4" event={"ID":"05676388-253f-42af-addc-8d66dcf95259","Type":"ContainerStarted","Data":"3ae1e7b1029c8ed877db2cfdd82b5cec022c706c3d54d54c72202393c4aa4be4"}
Sep 17 18:12:57.085502 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:57.085170    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4"
Sep 17 18:12:57.103738 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:12:57.103690    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4" podStartSLOduration=1.103651796 podStartE2EDuration="1.103651796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:12:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:57.103196057 +0000 UTC m=+2288.194502833" watchObservedRunningTime="2026-09-17 18:12:57.103651796 +0000 UTC m=+2288.194958562"
Sep 17 18:13:28.090837 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:28.090810    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4"
Sep 17 18:13:30.109958 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:30.109927    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8"]
Sep 17 18:13:30.113182 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:30.113165    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8"
Sep 17 18:13:30.117022 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:30.116987    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Sep 17 18:13:30.117224 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:30.117159    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 17 18:13:30.117326 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:30.117232    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 17 18:13:30.117398 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:30.117346    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-ttffv\""
Sep 17 18:13:30.124994 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:30.124969    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8"]
Sep 17 18:13:30.293830 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:30.293794    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c738c030-47c5-40e6-99e0-d074bbf7c473-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8\" (UID: \"c738c030-47c5-40e6-99e0-d074bbf7c473\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8"
Sep 17 18:13:30.294014 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:30.293855    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6jl6\" (UniqueName: \"kubernetes.io/projected/c738c030-47c5-40e6-99e0-d074bbf7c473-kube-api-access-n6jl6\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8\" (UID: \"c738c030-47c5-40e6-99e0-d074bbf7c473\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8"
Sep 17 18:13:30.294014 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:30.293889    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c738c030-47c5-40e6-99e0-d074bbf7c473-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8\" (UID: \"c738c030-47c5-40e6-99e0-d074bbf7c473\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8"
Sep 17 18:13:30.294014 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:30.293931    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c738c030-47c5-40e6-99e0-d074bbf7c473-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8\" (UID: \"c738c030-47c5-40e6-99e0-d074bbf7c473\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8"
Sep 17 18:13:30.294126 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:30.294019    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c738c030-47c5-40e6-99e0-d074bbf7c473-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8\" (UID: \"c738c030-47c5-40e6-99e0-d074bbf7c473\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8"
Sep 17 18:13:30.294126 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:30.294048    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c738c030-47c5-40e6-99e0-d074bbf7c473-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8\" (UID: \"c738c030-47c5-40e6-99e0-d074bbf7c473\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8"
Sep 17 18:13:30.394445 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:30.394366    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c738c030-47c5-40e6-99e0-d074bbf7c473-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8\" (UID: \"c738c030-47c5-40e6-99e0-d074bbf7c473\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8"
Sep 17 18:13:30.394445 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:30.394409    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c738c030-47c5-40e6-99e0-d074bbf7c473-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8\" (UID: \"c738c030-47c5-40e6-99e0-d074bbf7c473\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8"
Sep 17 18:13:30.394654 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:30.394449    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c738c030-47c5-40e6-99e0-d074bbf7c473-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8\" (UID: \"c738c030-47c5-40e6-99e0-d074bbf7c473\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8"
Sep 17 18:13:30.394654 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:30.394473    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c738c030-47c5-40e6-99e0-d074bbf7c473-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8\" (UID: \"c738c030-47c5-40e6-99e0-d074bbf7c473\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8"
Sep 17 18:13:30.394654 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:30.394508    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c738c030-47c5-40e6-99e0-d074bbf7c473-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8\" (UID: \"c738c030-47c5-40e6-99e0-d074bbf7c473\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8"
Sep 17 18:13:30.394654 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:30.394558    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6jl6\" (UniqueName: \"kubernetes.io/projected/c738c030-47c5-40e6-99e0-d074bbf7c473-kube-api-access-n6jl6\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8\" (UID: \"c738c030-47c5-40e6-99e0-d074bbf7c473\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8"
Sep 17 18:13:30.394906 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:30.394853    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c738c030-47c5-40e6-99e0-d074bbf7c473-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8\" (UID: \"c738c030-47c5-40e6-99e0-d074bbf7c473\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8"
Sep 17 18:13:30.394906 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:30.394892    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c738c030-47c5-40e6-99e0-d074bbf7c473-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8\" (UID: \"c738c030-47c5-40e6-99e0-d074bbf7c473\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8"
Sep 17 18:13:30.395001 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:30.394900    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c738c030-47c5-40e6-99e0-d074bbf7c473-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8\" (UID: \"c738c030-47c5-40e6-99e0-d074bbf7c473\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8"
Sep 17 18:13:30.396802 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:30.396782    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c738c030-47c5-40e6-99e0-d074bbf7c473-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8\" (UID: \"c738c030-47c5-40e6-99e0-d074bbf7c473\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8"
Sep 17 18:13:30.397097 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:30.397077    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c738c030-47c5-40e6-99e0-d074bbf7c473-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8\" (UID: \"c738c030-47c5-40e6-99e0-d074bbf7c473\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8"
Sep 17 18:13:30.405961 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:30.405937    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6jl6\" (UniqueName: \"kubernetes.io/projected/c738c030-47c5-40e6-99e0-d074bbf7c473-kube-api-access-n6jl6\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8\" (UID: \"c738c030-47c5-40e6-99e0-d074bbf7c473\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8"
Sep 17 18:13:30.424819 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:30.424798    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8"
Sep 17 18:13:30.699783 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:30.699757    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8"]
Sep 17 18:13:30.702176 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:13:30.702148    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc738c030_47c5_40e6_99e0_d074bbf7c473.slice/crio-88b1f4b3a80c84db08ed9878d086b0feef1a3ecab4b8758510aeac2976577675 WatchSource:0}: Error finding container 88b1f4b3a80c84db08ed9878d086b0feef1a3ecab4b8758510aeac2976577675: Status 404 returned error can't find the container with id 88b1f4b3a80c84db08ed9878d086b0feef1a3ecab4b8758510aeac2976577675
Sep 17 18:13:31.226732 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:31.226700    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8" event={"ID":"c738c030-47c5-40e6-99e0-d074bbf7c473","Type":"ContainerStarted","Data":"6574c2bcfd9f446e66aaae4aa603599fe172429f65ba49eaff6da632d47615e9"}
Sep 17 18:13:31.227159 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:31.226777    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8" event={"ID":"c738c030-47c5-40e6-99e0-d074bbf7c473","Type":"ContainerStarted","Data":"88b1f4b3a80c84db08ed9878d086b0feef1a3ecab4b8758510aeac2976577675"}
Sep 17 18:13:31.246802 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:31.246759    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8" podStartSLOduration=1.246740151 podStartE2EDuration="1.246740151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:13:31.244821878 +0000 UTC m=+2322.336128642" watchObservedRunningTime="2026-09-17 18:13:31.246740151 +0000 UTC m=+2322.338046920"
Sep 17 18:13:31.425045 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:31.425013    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8"
Sep 17 18:13:40.425601 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:40.425564    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8"
Sep 17 18:13:40.437802 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:40.437775    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8"
Sep 17 18:13:41.279648 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:41.279614    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8"
Sep 17 18:13:54.612439 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:54.612405    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8"]
Sep 17 18:13:54.620927 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:13:54.620892    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 18:13:54.620927 ip-10-0-141-30 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:13:54.620927 ip-10-0-141-30 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 18:13:54.620927 ip-10-0-141-30 kubenswrapper[2583]:  > containerID="6574c2bcfd9f446e66aaae4aa603599fe172429f65ba49eaff6da632d47615e9" cmd=["/bin/sleep","15"]
Sep 17 18:13:54.621172 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:54.620943    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8" message="" err=<
Sep 17 18:13:54.621172 ip-10-0-141-30 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:13:54.621172 ip-10-0-141-30 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 18:13:54.621172 ip-10-0-141-30 kubenswrapper[2583]:  >
Sep 17 18:13:54.621172 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:13:54.620955    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 18:13:54.621172 ip-10-0-141-30 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 18:13:54.621172 ip-10-0-141-30 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Sep 17 18:13:54.621172 ip-10-0-141-30 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8" podUID="c738c030-47c5-40e6-99e0-d074bbf7c473" containerName="main" containerID="cri-o://6574c2bcfd9f446e66aaae4aa603599fe172429f65ba49eaff6da632d47615e9"
Sep 17 18:13:54.621172 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:54.620998    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8" podUID="c738c030-47c5-40e6-99e0-d074bbf7c473" containerName="main" containerID="cri-o://6574c2bcfd9f446e66aaae4aa603599fe172429f65ba49eaff6da632d47615e9" gracePeriod=60
Sep 17 18:13:54.958641 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:54.958617    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8"
Sep 17 18:13:55.095334 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:55.095308    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c738c030-47c5-40e6-99e0-d074bbf7c473-home\") pod \"c738c030-47c5-40e6-99e0-d074bbf7c473\" (UID: \"c738c030-47c5-40e6-99e0-d074bbf7c473\") "
Sep 17 18:13:55.095521 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:55.095357    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6jl6\" (UniqueName: \"kubernetes.io/projected/c738c030-47c5-40e6-99e0-d074bbf7c473-kube-api-access-n6jl6\") pod \"c738c030-47c5-40e6-99e0-d074bbf7c473\" (UID: \"c738c030-47c5-40e6-99e0-d074bbf7c473\") "
Sep 17 18:13:55.095521 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:55.095377    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c738c030-47c5-40e6-99e0-d074bbf7c473-tls-certs\") pod \"c738c030-47c5-40e6-99e0-d074bbf7c473\" (UID: \"c738c030-47c5-40e6-99e0-d074bbf7c473\") "
Sep 17 18:13:55.095521 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:55.095406    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c738c030-47c5-40e6-99e0-d074bbf7c473-tmp-dir\") pod \"c738c030-47c5-40e6-99e0-d074bbf7c473\" (UID: \"c738c030-47c5-40e6-99e0-d074bbf7c473\") "
Sep 17 18:13:55.095521 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:55.095462    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c738c030-47c5-40e6-99e0-d074bbf7c473-model-cache\") pod \"c738c030-47c5-40e6-99e0-d074bbf7c473\" (UID: \"c738c030-47c5-40e6-99e0-d074bbf7c473\") "
Sep 17 18:13:55.095521 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:55.095490    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c738c030-47c5-40e6-99e0-d074bbf7c473-dshm\") pod \"c738c030-47c5-40e6-99e0-d074bbf7c473\" (UID: \"c738c030-47c5-40e6-99e0-d074bbf7c473\") "
Sep 17 18:13:55.095831 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:55.095607    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c738c030-47c5-40e6-99e0-d074bbf7c473-home" (OuterVolumeSpecName: "home") pod "c738c030-47c5-40e6-99e0-d074bbf7c473" (UID: "c738c030-47c5-40e6-99e0-d074bbf7c473"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:13:55.095831 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:55.095696    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c738c030-47c5-40e6-99e0-d074bbf7c473-model-cache" (OuterVolumeSpecName: "model-cache") pod "c738c030-47c5-40e6-99e0-d074bbf7c473" (UID: "c738c030-47c5-40e6-99e0-d074bbf7c473"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:13:55.095831 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:55.095725    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c738c030-47c5-40e6-99e0-d074bbf7c473-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c738c030-47c5-40e6-99e0-d074bbf7c473" (UID: "c738c030-47c5-40e6-99e0-d074bbf7c473"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:13:55.095831 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:55.095792    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c738c030-47c5-40e6-99e0-d074bbf7c473-model-cache\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:13:55.095831 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:55.095812    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c738c030-47c5-40e6-99e0-d074bbf7c473-home\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:13:55.095831 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:55.095826    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c738c030-47c5-40e6-99e0-d074bbf7c473-tmp-dir\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:13:55.097650 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:55.097620    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c738c030-47c5-40e6-99e0-d074bbf7c473-dshm" (OuterVolumeSpecName: "dshm") pod "c738c030-47c5-40e6-99e0-d074bbf7c473" (UID: "c738c030-47c5-40e6-99e0-d074bbf7c473"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:13:55.097650 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:55.097673    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c738c030-47c5-40e6-99e0-d074bbf7c473-kube-api-access-n6jl6" (OuterVolumeSpecName: "kube-api-access-n6jl6") pod "c738c030-47c5-40e6-99e0-d074bbf7c473" (UID: "c738c030-47c5-40e6-99e0-d074bbf7c473"). InnerVolumeSpecName "kube-api-access-n6jl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:13:55.097650 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:55.097679    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c738c030-47c5-40e6-99e0-d074bbf7c473-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c738c030-47c5-40e6-99e0-d074bbf7c473" (UID: "c738c030-47c5-40e6-99e0-d074bbf7c473"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:13:55.197193 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:55.197125    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c738c030-47c5-40e6-99e0-d074bbf7c473-dshm\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:13:55.197193 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:55.197150    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6jl6\" (UniqueName: \"kubernetes.io/projected/c738c030-47c5-40e6-99e0-d074bbf7c473-kube-api-access-n6jl6\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:13:55.197193 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:55.197160    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c738c030-47c5-40e6-99e0-d074bbf7c473-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:13:55.326044 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:55.326011    2583 generic.go:356] "Generic (PLEG): container finished" podID="c738c030-47c5-40e6-99e0-d074bbf7c473" containerID="6574c2bcfd9f446e66aaae4aa603599fe172429f65ba49eaff6da632d47615e9" exitCode=0
Sep 17 18:13:55.326213 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:55.326048    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8"
Sep 17 18:13:55.326213 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:55.326091    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8" event={"ID":"c738c030-47c5-40e6-99e0-d074bbf7c473","Type":"ContainerDied","Data":"6574c2bcfd9f446e66aaae4aa603599fe172429f65ba49eaff6da632d47615e9"}
Sep 17 18:13:55.326213 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:55.326142    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8" event={"ID":"c738c030-47c5-40e6-99e0-d074bbf7c473","Type":"ContainerDied","Data":"88b1f4b3a80c84db08ed9878d086b0feef1a3ecab4b8758510aeac2976577675"}
Sep 17 18:13:55.326213 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:55.326172    2583 scope.go:117] "RemoveContainer" containerID="6574c2bcfd9f446e66aaae4aa603599fe172429f65ba49eaff6da632d47615e9"
Sep 17 18:13:55.335116 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:55.335100    2583 scope.go:117] "RemoveContainer" containerID="6574c2bcfd9f446e66aaae4aa603599fe172429f65ba49eaff6da632d47615e9"
Sep 17 18:13:55.335358 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:13:55.335330    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6574c2bcfd9f446e66aaae4aa603599fe172429f65ba49eaff6da632d47615e9\": container with ID starting with 6574c2bcfd9f446e66aaae4aa603599fe172429f65ba49eaff6da632d47615e9 not found: ID does not exist" containerID="6574c2bcfd9f446e66aaae4aa603599fe172429f65ba49eaff6da632d47615e9"
Sep 17 18:13:55.335424 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:55.335370    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6574c2bcfd9f446e66aaae4aa603599fe172429f65ba49eaff6da632d47615e9"} err="failed to get container status \"6574c2bcfd9f446e66aaae4aa603599fe172429f65ba49eaff6da632d47615e9\": rpc error: code = NotFound desc = could not find container \"6574c2bcfd9f446e66aaae4aa603599fe172429f65ba49eaff6da632d47615e9\": container with ID starting with 6574c2bcfd9f446e66aaae4aa603599fe172429f65ba49eaff6da632d47615e9 not found: ID does not exist"
Sep 17 18:13:55.348814 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:55.348794    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8"]
Sep 17 18:13:55.356361 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:55.356340    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997qbbx8"]
Sep 17 18:13:57.045717 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:13:57.045682    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c738c030-47c5-40e6-99e0-d074bbf7c473" path="/var/lib/kubelet/pods/c738c030-47c5-40e6-99e0-d074bbf7c473/volumes"
Sep 17 18:14:19.013124 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.013088    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"]
Sep 17 18:14:19.013548 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.013518    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c738c030-47c5-40e6-99e0-d074bbf7c473" containerName="main"
Sep 17 18:14:19.013548 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.013534    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c738c030-47c5-40e6-99e0-d074bbf7c473" containerName="main"
Sep 17 18:14:19.013625 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.013587    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c738c030-47c5-40e6-99e0-d074bbf7c473" containerName="main"
Sep 17 18:14:19.020834 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.020810    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"
Sep 17 18:14:19.023425 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.023399    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:14:19.023572 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.023514    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 17 18:14:19.023572 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.023541    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 17 18:14:19.023572 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.023544    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-lrd2b\""
Sep 17 18:14:19.023788 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.023671    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 17 18:14:19.023788 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.023745    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Sep 17 18:14:19.030136 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.030106    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"]
Sep 17 18:14:19.192575 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.192541    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z\" (UID: \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"
Sep 17 18:14:19.192575 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.192578    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z\" (UID: \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"
Sep 17 18:14:19.192865 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.192599    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z\" (UID: \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"
Sep 17 18:14:19.192865 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.192652    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z\" (UID: \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"
Sep 17 18:14:19.192865 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.192681    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tgbj\" (UniqueName: \"kubernetes.io/projected/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-kube-api-access-2tgbj\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z\" (UID: \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"
Sep 17 18:14:19.192865 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.192727    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z\" (UID: \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"
Sep 17 18:14:19.192865 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.192790    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z\" (UID: \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"
Sep 17 18:14:19.193063 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.192872    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z\" (UID: \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"
Sep 17 18:14:19.293635 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.293540    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z\" (UID: \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"
Sep 17 18:14:19.293635 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.293585    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tgbj\" (UniqueName: \"kubernetes.io/projected/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-kube-api-access-2tgbj\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z\" (UID: \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"
Sep 17 18:14:19.293875 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.293715    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z\" (UID: \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"
Sep 17 18:14:19.293875 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.293772    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z\" (UID: \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"
Sep 17 18:14:19.293875 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.293815    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z\" (UID: \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"
Sep 17 18:14:19.294000 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.293873    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z\" (UID: \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"
Sep 17 18:14:19.294000 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.293904    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z\" (UID: \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"
Sep 17 18:14:19.294000 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.293930    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z\" (UID: \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"
Sep 17 18:14:19.294242 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.294114    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z\" (UID: \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"
Sep 17 18:14:19.294242 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.294168    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z\" (UID: \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"
Sep 17 18:14:19.294346 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.294255    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z\" (UID: \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"
Sep 17 18:14:19.294411 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.294392    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z\" (UID: \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"
Sep 17 18:14:19.294577 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.294559    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z\" (UID: \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"
Sep 17 18:14:19.295961 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.295941    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z\" (UID: \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"
Sep 17 18:14:19.296341 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.296321    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z\" (UID: \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"
Sep 17 18:14:19.302051 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.302027    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tgbj\" (UniqueName: \"kubernetes.io/projected/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-kube-api-access-2tgbj\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z\" (UID: \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"
Sep 17 18:14:19.332644 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.332621    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"
Sep 17 18:14:19.610014 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:19.609989    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"]
Sep 17 18:14:19.612294 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:14:19.612266    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a1d7dd_1ecc_4b16_8819_0aec760d8f7a.slice/crio-51d2473ce0b19f70f8d9e50dc3331c3322262b5143f9378b8c43e87e02e860ea WatchSource:0}: Error finding container 51d2473ce0b19f70f8d9e50dc3331c3322262b5143f9378b8c43e87e02e860ea: Status 404 returned error can't find the container with id 51d2473ce0b19f70f8d9e50dc3331c3322262b5143f9378b8c43e87e02e860ea
Sep 17 18:14:20.429330 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:20.429289    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z" event={"ID":"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a","Type":"ContainerStarted","Data":"316be94c89462acf67b815ffec41e724be591f3913797818938bc6a19694daac"}
Sep 17 18:14:20.429755 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:20.429351    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z" event={"ID":"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a","Type":"ContainerStarted","Data":"51d2473ce0b19f70f8d9e50dc3331c3322262b5143f9378b8c43e87e02e860ea"}
Sep 17 18:14:22.439088 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:22.439054    2583 generic.go:356] "Generic (PLEG): container finished" podID="c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a" containerID="316be94c89462acf67b815ffec41e724be591f3913797818938bc6a19694daac" exitCode=0
Sep 17 18:14:22.439492 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:22.439122    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z" event={"ID":"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a","Type":"ContainerDied","Data":"316be94c89462acf67b815ffec41e724be591f3913797818938bc6a19694daac"}
Sep 17 18:14:23.445311 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:23.445275    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z" event={"ID":"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a","Type":"ContainerStarted","Data":"a05649ae9977efa9163716c2fc5640337b70ba251feb694df26e7284c898c1cf"}
Sep 17 18:14:23.471392 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:23.471329    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z" podStartSLOduration=3.349169875 podStartE2EDuration="5.471313649s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.122143774s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:14:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:14:23.468990492 +0000 UTC m=+2374.560297275" watchObservedRunningTime="2026-09-17 18:14:23.471313649 +0000 UTC m=+2374.562620414"
Sep 17 18:14:24.333009 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:24.332973    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"
Sep 17 18:14:29.332922 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:29.332885    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"
Sep 17 18:14:29.334501 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:29.334472    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z" podUID="c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 17 18:14:39.333632 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:39.333587    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z" podUID="c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 17 18:14:49.334131 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:49.334080    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z" podUID="c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 17 18:14:59.333647 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:14:59.333598    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z" podUID="c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 17 18:15:09.333132 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:15:09.333024    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z" podUID="c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 17 18:15:19.333182 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:15:19.333131    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z" podUID="c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 17 18:15:25.323547 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:15:25.323506    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4"]
Sep 17 18:15:28.090730 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:15:28.090684    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4" podUID="05676388-253f-42af-addc-8d66dcf95259" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:15:29.333589 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:15:29.333546    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z" podUID="c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 17 18:15:38.091277 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:15:38.091240    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4" podUID="05676388-253f-42af-addc-8d66dcf95259" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:15:39.333078 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:15:39.333034    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z" podUID="c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 17 18:15:40.344126 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:15:40.344077    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4" podUID="05676388-253f-42af-addc-8d66dcf95259" containerName="main" containerID="cri-o://1d0ef7575189dada8e0e1a8890fbeed8e3d7b53280f6af9d2cad29990c89ad51" gracePeriod=45
Sep 17 18:15:48.091048 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:15:48.091010    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4" podUID="05676388-253f-42af-addc-8d66dcf95259" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:15:48.093610 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:15:48.091123    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4"
Sep 17 18:15:49.333609 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:15:49.333560    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z" podUID="c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 17 18:15:58.091278 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:15:58.091240    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4" podUID="05676388-253f-42af-addc-8d66dcf95259" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:15:59.334087 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:15:59.334042    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z" podUID="c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 17 18:16:08.091121 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:08.091074    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4" podUID="05676388-253f-42af-addc-8d66dcf95259" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:16:09.333778 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:09.333738    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z" podUID="c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 17 18:16:10.611306 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:10.611235    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4"
Sep 17 18:16:10.661623 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:10.661589    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffcjg\" (UniqueName: \"kubernetes.io/projected/05676388-253f-42af-addc-8d66dcf95259-kube-api-access-ffcjg\") pod \"05676388-253f-42af-addc-8d66dcf95259\" (UID: \"05676388-253f-42af-addc-8d66dcf95259\") "
Sep 17 18:16:10.661820 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:10.661701    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05676388-253f-42af-addc-8d66dcf95259-tls-certs\") pod \"05676388-253f-42af-addc-8d66dcf95259\" (UID: \"05676388-253f-42af-addc-8d66dcf95259\") "
Sep 17 18:16:10.663964 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:10.663925    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05676388-253f-42af-addc-8d66dcf95259-kube-api-access-ffcjg" (OuterVolumeSpecName: "kube-api-access-ffcjg") pod "05676388-253f-42af-addc-8d66dcf95259" (UID: "05676388-253f-42af-addc-8d66dcf95259"). InnerVolumeSpecName "kube-api-access-ffcjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:16:10.664272 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:10.664242    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05676388-253f-42af-addc-8d66dcf95259-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "05676388-253f-42af-addc-8d66dcf95259" (UID: "05676388-253f-42af-addc-8d66dcf95259"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:16:10.762946 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:10.762902    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05676388-253f-42af-addc-8d66dcf95259-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:16:10.762946 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:10.762945    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffcjg\" (UniqueName: \"kubernetes.io/projected/05676388-253f-42af-addc-8d66dcf95259-kube-api-access-ffcjg\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:16:10.933752 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:10.933644    2583 generic.go:356] "Generic (PLEG): container finished" podID="05676388-253f-42af-addc-8d66dcf95259" containerID="1d0ef7575189dada8e0e1a8890fbeed8e3d7b53280f6af9d2cad29990c89ad51" exitCode=0
Sep 17 18:16:10.933752 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:10.933705    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4"
Sep 17 18:16:10.933752 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:10.933715    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4" event={"ID":"05676388-253f-42af-addc-8d66dcf95259","Type":"ContainerDied","Data":"1d0ef7575189dada8e0e1a8890fbeed8e3d7b53280f6af9d2cad29990c89ad51"}
Sep 17 18:16:10.933982 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:10.933782    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4" event={"ID":"05676388-253f-42af-addc-8d66dcf95259","Type":"ContainerDied","Data":"3ae1e7b1029c8ed877db2cfdd82b5cec022c706c3d54d54c72202393c4aa4be4"}
Sep 17 18:16:10.933982 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:10.933803    2583 scope.go:117] "RemoveContainer" containerID="1d0ef7575189dada8e0e1a8890fbeed8e3d7b53280f6af9d2cad29990c89ad51"
Sep 17 18:16:10.942951 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:10.942925    2583 scope.go:117] "RemoveContainer" containerID="1d0ef7575189dada8e0e1a8890fbeed8e3d7b53280f6af9d2cad29990c89ad51"
Sep 17 18:16:10.943210 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:16:10.943190    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d0ef7575189dada8e0e1a8890fbeed8e3d7b53280f6af9d2cad29990c89ad51\": container with ID starting with 1d0ef7575189dada8e0e1a8890fbeed8e3d7b53280f6af9d2cad29990c89ad51 not found: ID does not exist" containerID="1d0ef7575189dada8e0e1a8890fbeed8e3d7b53280f6af9d2cad29990c89ad51"
Sep 17 18:16:10.943284 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:10.943217    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d0ef7575189dada8e0e1a8890fbeed8e3d7b53280f6af9d2cad29990c89ad51"} err="failed to get container status \"1d0ef7575189dada8e0e1a8890fbeed8e3d7b53280f6af9d2cad29990c89ad51\": rpc error: code = NotFound desc = could not find container \"1d0ef7575189dada8e0e1a8890fbeed8e3d7b53280f6af9d2cad29990c89ad51\": container with ID starting with 1d0ef7575189dada8e0e1a8890fbeed8e3d7b53280f6af9d2cad29990c89ad51 not found: ID does not exist"
Sep 17 18:16:10.955306 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:10.955279    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4"]
Sep 17 18:16:10.959095 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:10.959072    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schevwzs4"]
Sep 17 18:16:11.040963 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:11.040923    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05676388-253f-42af-addc-8d66dcf95259" path="/var/lib/kubelet/pods/05676388-253f-42af-addc-8d66dcf95259/volumes"
Sep 17 18:16:19.343444 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:19.343407    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"
Sep 17 18:16:19.351166 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:19.351143    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"
Sep 17 18:16:20.457563 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:20.457526    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm"]
Sep 17 18:16:20.458095 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:20.457971    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05676388-253f-42af-addc-8d66dcf95259" containerName="main"
Sep 17 18:16:20.458095 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:20.457991    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="05676388-253f-42af-addc-8d66dcf95259" containerName="main"
Sep 17 18:16:20.458095 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:20.458091    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="05676388-253f-42af-addc-8d66dcf95259" containerName="main"
Sep 17 18:16:20.459994 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:20.459972    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm"
Sep 17 18:16:20.463719 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:20.463698    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 17 18:16:20.463834 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:20.463755    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 17 18:16:20.463834 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:20.463808    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 17 18:16:20.464023 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:20.463944    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-pg56v\""
Sep 17 18:16:20.464232 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:20.464195    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-grhfx\""
Sep 17 18:16:20.470172 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:20.470152    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm"]
Sep 17 18:16:20.544845 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:20.544811    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2822600-59ff-4121-b209-d10d65ac2c71-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm\" (UID: \"b2822600-59ff-4121-b209-d10d65ac2c71\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm"
Sep 17 18:16:20.544997 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:20.544925    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr7jg\" (UniqueName: \"kubernetes.io/projected/b2822600-59ff-4121-b209-d10d65ac2c71-kube-api-access-fr7jg\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm\" (UID: \"b2822600-59ff-4121-b209-d10d65ac2c71\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm"
Sep 17 18:16:20.645777 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:20.645747    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr7jg\" (UniqueName: \"kubernetes.io/projected/b2822600-59ff-4121-b209-d10d65ac2c71-kube-api-access-fr7jg\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm\" (UID: \"b2822600-59ff-4121-b209-d10d65ac2c71\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm"
Sep 17 18:16:20.645946 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:20.645784    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2822600-59ff-4121-b209-d10d65ac2c71-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm\" (UID: \"b2822600-59ff-4121-b209-d10d65ac2c71\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm"
Sep 17 18:16:20.648273 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:20.648244    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2822600-59ff-4121-b209-d10d65ac2c71-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm\" (UID: \"b2822600-59ff-4121-b209-d10d65ac2c71\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm"
Sep 17 18:16:20.655977 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:20.655954    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr7jg\" (UniqueName: \"kubernetes.io/projected/b2822600-59ff-4121-b209-d10d65ac2c71-kube-api-access-fr7jg\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm\" (UID: \"b2822600-59ff-4121-b209-d10d65ac2c71\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm"
Sep 17 18:16:20.770768 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:20.770692    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm"
Sep 17 18:16:21.041316 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:21.041238    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm"]
Sep 17 18:16:21.044517 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:16:21.044492    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2822600_59ff_4121_b209_d10d65ac2c71.slice/crio-b6fba3dd5f14a148696e81bb46b919bb58961a72acf058f4635784b45fdda63c WatchSource:0}: Error finding container b6fba3dd5f14a148696e81bb46b919bb58961a72acf058f4635784b45fdda63c: Status 404 returned error can't find the container with id b6fba3dd5f14a148696e81bb46b919bb58961a72acf058f4635784b45fdda63c
Sep 17 18:16:21.986138 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:21.986097    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm" event={"ID":"b2822600-59ff-4121-b209-d10d65ac2c71","Type":"ContainerStarted","Data":"07996bf405c50a214fa3aca31005da8a39481d6fb3a9eaf6e125a0293ccf979e"}
Sep 17 18:16:21.986627 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:21.986151    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm" event={"ID":"b2822600-59ff-4121-b209-d10d65ac2c71","Type":"ContainerStarted","Data":"b6fba3dd5f14a148696e81bb46b919bb58961a72acf058f4635784b45fdda63c"}
Sep 17 18:16:21.986627 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:21.986231    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm"
Sep 17 18:16:22.015107 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:22.015058    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm" podStartSLOduration=2.015045068 podStartE2EDuration="2.015045068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:22.01335796 +0000 UTC m=+2493.104664737" watchObservedRunningTime="2026-09-17 18:16:22.015045068 +0000 UTC m=+2493.106351833"
Sep 17 18:16:32.504233 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:32.504142    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"]
Sep 17 18:16:47.526474 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:47.526407    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z" podUID="c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a" containerName="main" containerID="cri-o://a05649ae9977efa9163716c2fc5640337b70ba251feb694df26e7284c898c1cf" gracePeriod=45
Sep 17 18:16:48.271385 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:48.271359    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"
Sep 17 18:16:48.375636 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:48.375607    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-tls-certs\") pod \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\" (UID: \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\") "
Sep 17 18:16:48.375829 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:48.375650    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-cabundle-cert\") pod \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\" (UID: \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\") "
Sep 17 18:16:48.375829 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:48.375707    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-kserve-provision-location\") pod \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\" (UID: \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\") "
Sep 17 18:16:48.375829 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:48.375733    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-home\") pod \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\" (UID: \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\") "
Sep 17 18:16:48.375829 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:48.375755    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tgbj\" (UniqueName: \"kubernetes.io/projected/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-kube-api-access-2tgbj\") pod \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\" (UID: \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\") "
Sep 17 18:16:48.375829 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:48.375803    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-tmp-dir\") pod \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\" (UID: \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\") "
Sep 17 18:16:48.376111 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:48.375870    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-dshm\") pod \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\" (UID: \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\") "
Sep 17 18:16:48.376111 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:48.375905    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-model-cache\") pod \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\" (UID: \"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a\") "
Sep 17 18:16:48.376311 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:48.376286    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-model-cache" (OuterVolumeSpecName: "model-cache") pod "c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a" (UID: "c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:16:48.376498 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:48.376408    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a" (UID: "c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:16:48.376602 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:48.376479    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-home" (OuterVolumeSpecName: "home") pod "c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a" (UID: "c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:16:48.378303 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:48.378277    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a" (UID: "c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:16:48.378388 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:48.378279    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-dshm" (OuterVolumeSpecName: "dshm") pod "c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a" (UID: "c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:16:48.378388 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:48.378313    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-kube-api-access-2tgbj" (OuterVolumeSpecName: "kube-api-access-2tgbj") pod "c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a" (UID: "c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a"). InnerVolumeSpecName "kube-api-access-2tgbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:16:48.388297 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:48.388271    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a" (UID: "c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:16:48.404473 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:48.404448    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a" (UID: "c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:16:48.476578 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:48.476550    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:16:48.476578 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:48.476575    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-cabundle-cert\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:16:48.476762 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:48.476584    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-kserve-provision-location\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:16:48.476762 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:48.476593    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-home\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:16:48.476762 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:48.476601    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tgbj\" (UniqueName: \"kubernetes.io/projected/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-kube-api-access-2tgbj\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:16:48.476762 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:48.476609    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-tmp-dir\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:16:48.476762 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:48.476616    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-dshm\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:16:48.476762 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:48.476624    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a-model-cache\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:16:49.097164 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:49.097126    2583 generic.go:356] "Generic (PLEG): container finished" podID="c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a" containerID="a05649ae9977efa9163716c2fc5640337b70ba251feb694df26e7284c898c1cf" exitCode=0
Sep 17 18:16:49.097590 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:49.097175    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"
Sep 17 18:16:49.097590 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:49.097211    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z" event={"ID":"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a","Type":"ContainerDied","Data":"a05649ae9977efa9163716c2fc5640337b70ba251feb694df26e7284c898c1cf"}
Sep 17 18:16:49.097590 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:49.097266    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z" event={"ID":"c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a","Type":"ContainerDied","Data":"51d2473ce0b19f70f8d9e50dc3331c3322262b5143f9378b8c43e87e02e860ea"}
Sep 17 18:16:49.097590 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:49.097298    2583 scope.go:117] "RemoveContainer" containerID="a05649ae9977efa9163716c2fc5640337b70ba251feb694df26e7284c898c1cf"
Sep 17 18:16:49.105977 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:49.105959    2583 scope.go:117] "RemoveContainer" containerID="316be94c89462acf67b815ffec41e724be591f3913797818938bc6a19694daac"
Sep 17 18:16:49.113716 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:49.113698    2583 scope.go:117] "RemoveContainer" containerID="a05649ae9977efa9163716c2fc5640337b70ba251feb694df26e7284c898c1cf"
Sep 17 18:16:49.113998 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:16:49.113976    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a05649ae9977efa9163716c2fc5640337b70ba251feb694df26e7284c898c1cf\": container with ID starting with a05649ae9977efa9163716c2fc5640337b70ba251feb694df26e7284c898c1cf not found: ID does not exist" containerID="a05649ae9977efa9163716c2fc5640337b70ba251feb694df26e7284c898c1cf"
Sep 17 18:16:49.114064 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:49.114006    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a05649ae9977efa9163716c2fc5640337b70ba251feb694df26e7284c898c1cf"} err="failed to get container status \"a05649ae9977efa9163716c2fc5640337b70ba251feb694df26e7284c898c1cf\": rpc error: code = NotFound desc = could not find container \"a05649ae9977efa9163716c2fc5640337b70ba251feb694df26e7284c898c1cf\": container with ID starting with a05649ae9977efa9163716c2fc5640337b70ba251feb694df26e7284c898c1cf not found: ID does not exist"
Sep 17 18:16:49.114064 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:49.114023    2583 scope.go:117] "RemoveContainer" containerID="316be94c89462acf67b815ffec41e724be591f3913797818938bc6a19694daac"
Sep 17 18:16:49.114293 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:16:49.114278    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"316be94c89462acf67b815ffec41e724be591f3913797818938bc6a19694daac\": container with ID starting with 316be94c89462acf67b815ffec41e724be591f3913797818938bc6a19694daac not found: ID does not exist" containerID="316be94c89462acf67b815ffec41e724be591f3913797818938bc6a19694daac"
Sep 17 18:16:49.114336 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:49.114296    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"316be94c89462acf67b815ffec41e724be591f3913797818938bc6a19694daac"} err="failed to get container status \"316be94c89462acf67b815ffec41e724be591f3913797818938bc6a19694daac\": rpc error: code = NotFound desc = could not find container \"316be94c89462acf67b815ffec41e724be591f3913797818938bc6a19694daac\": container with ID starting with 316be94c89462acf67b815ffec41e724be591f3913797818938bc6a19694daac not found: ID does not exist"
Sep 17 18:16:49.125627 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:49.125601    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"]
Sep 17 18:16:49.132898 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:49.132877    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5b7dd69d7996l2z"]
Sep 17 18:16:51.040335 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:51.040302    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a" path="/var/lib/kubelet/pods/c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a/volumes"
Sep 17 18:16:52.992780 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:16:52.992754    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm"
Sep 17 18:17:06.456966 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:17:06.456930    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb"]
Sep 17 18:17:06.457447 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:17:06.457252    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a" containerName="main"
Sep 17 18:17:06.457447 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:17:06.457266    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a" containerName="main"
Sep 17 18:17:06.457447 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:17:06.457281    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a" containerName="storage-initializer"
Sep 17 18:17:06.457447 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:17:06.457287    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a" containerName="storage-initializer"
Sep 17 18:17:06.457447 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:17:06.457336    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a1d7dd-1ecc-4b16-8819-0aec760d8f7a" containerName="main"
Sep 17 18:17:06.460262 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:17:06.460242    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb"
Sep 17 18:17:06.463555 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:17:06.463529    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-2ptdb\""
Sep 17 18:17:06.463766 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:17:06.463741    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 17 18:17:06.470962 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:17:06.470939    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb"]
Sep 17 18:17:06.509667 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:17:06.509641    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0dfe9c4-1282-4d10-b35b-81e6219c5c1e-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb\" (UID: \"f0dfe9c4-1282-4d10-b35b-81e6219c5c1e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb"
Sep 17 18:17:06.509769 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:17:06.509716    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmgsv\" (UniqueName: \"kubernetes.io/projected/f0dfe9c4-1282-4d10-b35b-81e6219c5c1e-kube-api-access-mmgsv\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb\" (UID: \"f0dfe9c4-1282-4d10-b35b-81e6219c5c1e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb"
Sep 17 18:17:06.610602 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:17:06.610572    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0dfe9c4-1282-4d10-b35b-81e6219c5c1e-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb\" (UID: \"f0dfe9c4-1282-4d10-b35b-81e6219c5c1e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb"
Sep 17 18:17:06.610739 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:17:06.610617    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmgsv\" (UniqueName: \"kubernetes.io/projected/f0dfe9c4-1282-4d10-b35b-81e6219c5c1e-kube-api-access-mmgsv\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb\" (UID: \"f0dfe9c4-1282-4d10-b35b-81e6219c5c1e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb"
Sep 17 18:17:06.613146 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:17:06.613123    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0dfe9c4-1282-4d10-b35b-81e6219c5c1e-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb\" (UID: \"f0dfe9c4-1282-4d10-b35b-81e6219c5c1e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb"
Sep 17 18:17:06.619109 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:17:06.619082    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmgsv\" (UniqueName: \"kubernetes.io/projected/f0dfe9c4-1282-4d10-b35b-81e6219c5c1e-kube-api-access-mmgsv\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb\" (UID: \"f0dfe9c4-1282-4d10-b35b-81e6219c5c1e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb"
Sep 17 18:17:06.771561 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:17:06.771483    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb"
Sep 17 18:17:07.040948 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:17:07.040820    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb"]
Sep 17 18:17:07.043420 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:17:07.043398    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0dfe9c4_1282_4d10_b35b_81e6219c5c1e.slice/crio-27a2179173f7a77e022d9788df5ab1ceaee1e3cb2e6dd25a59e4972d0f212147 WatchSource:0}: Error finding container 27a2179173f7a77e022d9788df5ab1ceaee1e3cb2e6dd25a59e4972d0f212147: Status 404 returned error can't find the container with id 27a2179173f7a77e022d9788df5ab1ceaee1e3cb2e6dd25a59e4972d0f212147
Sep 17 18:17:07.174858 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:17:07.174822    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb" event={"ID":"f0dfe9c4-1282-4d10-b35b-81e6219c5c1e","Type":"ContainerStarted","Data":"9b6b4f7b0ba07c8d445d0c4db7a8914da761a12b7fe491aabf16fbd10f45938c"}
Sep 17 18:17:07.175023 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:17:07.174884    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb"
Sep 17 18:17:07.175023 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:17:07.174905    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb" event={"ID":"f0dfe9c4-1282-4d10-b35b-81e6219c5c1e","Type":"ContainerStarted","Data":"27a2179173f7a77e022d9788df5ab1ceaee1e3cb2e6dd25a59e4972d0f212147"}
Sep 17 18:17:07.208168 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:17:07.208124    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb" podStartSLOduration=1.208107448 podStartE2EDuration="1.208107448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:17:07.205956952 +0000 UTC m=+2538.297263730" watchObservedRunningTime="2026-09-17 18:17:07.208107448 +0000 UTC m=+2538.299414212"
Sep 17 18:17:38.181389 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:17:38.181354    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb"
Sep 17 18:17:56.950225 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:17:56.950185    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb"]
Sep 17 18:17:58.181934 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:17:58.181887    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb" podUID="f0dfe9c4-1282-4d10-b35b-81e6219c5c1e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:18:08.181519 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:18:08.181433    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb" podUID="f0dfe9c4-1282-4d10-b35b-81e6219c5c1e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:18:11.971726 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:18:11.971678    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb" podUID="f0dfe9c4-1282-4d10-b35b-81e6219c5c1e" containerName="main" containerID="cri-o://9b6b4f7b0ba07c8d445d0c4db7a8914da761a12b7fe491aabf16fbd10f45938c" gracePeriod=45
Sep 17 18:18:18.181654 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:18:18.181616    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb" podUID="f0dfe9c4-1282-4d10-b35b-81e6219c5c1e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:18:18.182158 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:18:18.181767    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb"
Sep 17 18:18:28.181373 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:18:28.181335    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb" podUID="f0dfe9c4-1282-4d10-b35b-81e6219c5c1e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:18:38.181435 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:18:38.181398    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb" podUID="f0dfe9c4-1282-4d10-b35b-81e6219c5c1e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:18:42.212612 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:18:42.212584    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb"
Sep 17 18:18:42.275603 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:18:42.275524    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0dfe9c4-1282-4d10-b35b-81e6219c5c1e-tls-certs\") pod \"f0dfe9c4-1282-4d10-b35b-81e6219c5c1e\" (UID: \"f0dfe9c4-1282-4d10-b35b-81e6219c5c1e\") "
Sep 17 18:18:42.275603 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:18:42.275562    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmgsv\" (UniqueName: \"kubernetes.io/projected/f0dfe9c4-1282-4d10-b35b-81e6219c5c1e-kube-api-access-mmgsv\") pod \"f0dfe9c4-1282-4d10-b35b-81e6219c5c1e\" (UID: \"f0dfe9c4-1282-4d10-b35b-81e6219c5c1e\") "
Sep 17 18:18:42.277811 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:18:42.277778    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0dfe9c4-1282-4d10-b35b-81e6219c5c1e-kube-api-access-mmgsv" (OuterVolumeSpecName: "kube-api-access-mmgsv") pod "f0dfe9c4-1282-4d10-b35b-81e6219c5c1e" (UID: "f0dfe9c4-1282-4d10-b35b-81e6219c5c1e"). InnerVolumeSpecName "kube-api-access-mmgsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:18:42.277811 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:18:42.277782    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0dfe9c4-1282-4d10-b35b-81e6219c5c1e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f0dfe9c4-1282-4d10-b35b-81e6219c5c1e" (UID: "f0dfe9c4-1282-4d10-b35b-81e6219c5c1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:18:42.376678 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:18:42.376626    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0dfe9c4-1282-4d10-b35b-81e6219c5c1e-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:18:42.376678 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:18:42.376656    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmgsv\" (UniqueName: \"kubernetes.io/projected/f0dfe9c4-1282-4d10-b35b-81e6219c5c1e-kube-api-access-mmgsv\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:18:42.593015 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:18:42.592933    2583 generic.go:356] "Generic (PLEG): container finished" podID="f0dfe9c4-1282-4d10-b35b-81e6219c5c1e" containerID="9b6b4f7b0ba07c8d445d0c4db7a8914da761a12b7fe491aabf16fbd10f45938c" exitCode=0
Sep 17 18:18:42.593015 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:18:42.592979    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb"
Sep 17 18:18:42.593204 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:18:42.592984    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb" event={"ID":"f0dfe9c4-1282-4d10-b35b-81e6219c5c1e","Type":"ContainerDied","Data":"9b6b4f7b0ba07c8d445d0c4db7a8914da761a12b7fe491aabf16fbd10f45938c"}
Sep 17 18:18:42.593204 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:18:42.593106    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb" event={"ID":"f0dfe9c4-1282-4d10-b35b-81e6219c5c1e","Type":"ContainerDied","Data":"27a2179173f7a77e022d9788df5ab1ceaee1e3cb2e6dd25a59e4972d0f212147"}
Sep 17 18:18:42.593204 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:18:42.593130    2583 scope.go:117] "RemoveContainer" containerID="9b6b4f7b0ba07c8d445d0c4db7a8914da761a12b7fe491aabf16fbd10f45938c"
Sep 17 18:18:42.603876 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:18:42.603854    2583 scope.go:117] "RemoveContainer" containerID="9b6b4f7b0ba07c8d445d0c4db7a8914da761a12b7fe491aabf16fbd10f45938c"
Sep 17 18:18:42.604179 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:18:42.604161    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b6b4f7b0ba07c8d445d0c4db7a8914da761a12b7fe491aabf16fbd10f45938c\": container with ID starting with 9b6b4f7b0ba07c8d445d0c4db7a8914da761a12b7fe491aabf16fbd10f45938c not found: ID does not exist" containerID="9b6b4f7b0ba07c8d445d0c4db7a8914da761a12b7fe491aabf16fbd10f45938c"
Sep 17 18:18:42.604293 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:18:42.604187    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b6b4f7b0ba07c8d445d0c4db7a8914da761a12b7fe491aabf16fbd10f45938c"} err="failed to get container status \"9b6b4f7b0ba07c8d445d0c4db7a8914da761a12b7fe491aabf16fbd10f45938c\": rpc error: code = NotFound desc = could not find container \"9b6b4f7b0ba07c8d445d0c4db7a8914da761a12b7fe491aabf16fbd10f45938c\": container with ID starting with 9b6b4f7b0ba07c8d445d0c4db7a8914da761a12b7fe491aabf16fbd10f45938c not found: ID does not exist"
Sep 17 18:18:42.614120 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:18:42.614092    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb"]
Sep 17 18:18:42.620090 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:18:42.620066    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schekk5wb"]
Sep 17 18:18:43.039891 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:18:43.039855    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0dfe9c4-1282-4d10-b35b-81e6219c5c1e" path="/var/lib/kubelet/pods/f0dfe9c4-1282-4d10-b35b-81e6219c5c1e/volumes"
Sep 17 18:18:47.561433 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:18:47.561403    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm"]
Sep 17 18:18:52.992675 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:18:52.992626    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm" podUID="b2822600-59ff-4121-b209-d10d65ac2c71" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:19:02.580436 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:02.580362    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm" podUID="b2822600-59ff-4121-b209-d10d65ac2c71" containerName="main" containerID="cri-o://07996bf405c50a214fa3aca31005da8a39481d6fb3a9eaf6e125a0293ccf979e" gracePeriod=45
Sep 17 18:19:02.992805 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:02.992768    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm" podUID="b2822600-59ff-4121-b209-d10d65ac2c71" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:19:12.992837 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:12.992795    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm" podUID="b2822600-59ff-4121-b209-d10d65ac2c71" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:19:12.993240 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:12.992901    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm"
Sep 17 18:19:22.992400 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:22.992357    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm" podUID="b2822600-59ff-4121-b209-d10d65ac2c71" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:19:32.800960 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:32.800922    2583 generic.go:356] "Generic (PLEG): container finished" podID="b2822600-59ff-4121-b209-d10d65ac2c71" containerID="07996bf405c50a214fa3aca31005da8a39481d6fb3a9eaf6e125a0293ccf979e" exitCode=0
Sep 17 18:19:32.801354 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:32.800986    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm" event={"ID":"b2822600-59ff-4121-b209-d10d65ac2c71","Type":"ContainerDied","Data":"07996bf405c50a214fa3aca31005da8a39481d6fb3a9eaf6e125a0293ccf979e"}
Sep 17 18:19:32.831620 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:32.831598    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm"
Sep 17 18:19:32.969643 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:32.969612    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2822600-59ff-4121-b209-d10d65ac2c71-tls-certs\") pod \"b2822600-59ff-4121-b209-d10d65ac2c71\" (UID: \"b2822600-59ff-4121-b209-d10d65ac2c71\") "
Sep 17 18:19:32.969806 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:32.969720    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fr7jg\" (UniqueName: \"kubernetes.io/projected/b2822600-59ff-4121-b209-d10d65ac2c71-kube-api-access-fr7jg\") pod \"b2822600-59ff-4121-b209-d10d65ac2c71\" (UID: \"b2822600-59ff-4121-b209-d10d65ac2c71\") "
Sep 17 18:19:32.971687 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:32.971647    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2822600-59ff-4121-b209-d10d65ac2c71-kube-api-access-fr7jg" (OuterVolumeSpecName: "kube-api-access-fr7jg") pod "b2822600-59ff-4121-b209-d10d65ac2c71" (UID: "b2822600-59ff-4121-b209-d10d65ac2c71"). InnerVolumeSpecName "kube-api-access-fr7jg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:19:32.971797 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:32.971736    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2822600-59ff-4121-b209-d10d65ac2c71-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b2822600-59ff-4121-b209-d10d65ac2c71" (UID: "b2822600-59ff-4121-b209-d10d65ac2c71"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:19:33.070861 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:33.070839    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr7jg\" (UniqueName: \"kubernetes.io/projected/b2822600-59ff-4121-b209-d10d65ac2c71-kube-api-access-fr7jg\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:19:33.070861 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:33.070860    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2822600-59ff-4121-b209-d10d65ac2c71-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:19:33.806787 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:33.806747    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm" event={"ID":"b2822600-59ff-4121-b209-d10d65ac2c71","Type":"ContainerDied","Data":"b6fba3dd5f14a148696e81bb46b919bb58961a72acf058f4635784b45fdda63c"}
Sep 17 18:19:33.806787 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:33.806778    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm"
Sep 17 18:19:33.807279 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:33.806823    2583 scope.go:117] "RemoveContainer" containerID="07996bf405c50a214fa3aca31005da8a39481d6fb3a9eaf6e125a0293ccf979e"
Sep 17 18:19:33.822632 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:33.822609    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm"]
Sep 17 18:19:33.826328 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:33.826299    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche4d4dm"]
Sep 17 18:19:35.039502 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:35.039468    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2822600-59ff-4121-b209-d10d65ac2c71" path="/var/lib/kubelet/pods/b2822600-59ff-4121-b209-d10d65ac2c71/volumes"
Sep 17 18:19:49.125386 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.125358    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"]
Sep 17 18:19:49.125910 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.125686    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2822600-59ff-4121-b209-d10d65ac2c71" containerName="main"
Sep 17 18:19:49.125910 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.125701    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2822600-59ff-4121-b209-d10d65ac2c71" containerName="main"
Sep 17 18:19:49.125910 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.125715    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0dfe9c4-1282-4d10-b35b-81e6219c5c1e" containerName="main"
Sep 17 18:19:49.125910 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.125720    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0dfe9c4-1282-4d10-b35b-81e6219c5c1e" containerName="main"
Sep 17 18:19:49.125910 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.125779    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0dfe9c4-1282-4d10-b35b-81e6219c5c1e" containerName="main"
Sep 17 18:19:49.125910 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.125790    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2822600-59ff-4121-b209-d10d65ac2c71" containerName="main"
Sep 17 18:19:49.128521 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.128504    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"
Sep 17 18:19:49.131226 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.131204    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 17 18:19:49.131355 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.131206    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Sep 17 18:19:49.131355 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.131208    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-b8nrd\""
Sep 17 18:19:49.131355 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.131294    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 17 18:19:49.131355 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.131311    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 17 18:19:49.131573 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.131365    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:19:49.142820 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.142791    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"]
Sep 17 18:19:49.290295 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.290253    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2gdc\" (UniqueName: \"kubernetes.io/projected/0af8544c-568c-4931-ba63-6aa8450fab29-kube-api-access-x2gdc\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd\" (UID: \"0af8544c-568c-4931-ba63-6aa8450fab29\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"
Sep 17 18:19:49.290295 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.290294    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af8544c-568c-4931-ba63-6aa8450fab29-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd\" (UID: \"0af8544c-568c-4931-ba63-6aa8450fab29\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"
Sep 17 18:19:49.290730 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.290324    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af8544c-568c-4931-ba63-6aa8450fab29-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd\" (UID: \"0af8544c-568c-4931-ba63-6aa8450fab29\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"
Sep 17 18:19:49.290730 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.290406    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0af8544c-568c-4931-ba63-6aa8450fab29-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd\" (UID: \"0af8544c-568c-4931-ba63-6aa8450fab29\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"
Sep 17 18:19:49.290730 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.290448    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af8544c-568c-4931-ba63-6aa8450fab29-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd\" (UID: \"0af8544c-568c-4931-ba63-6aa8450fab29\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"
Sep 17 18:19:49.290730 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.290468    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af8544c-568c-4931-ba63-6aa8450fab29-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd\" (UID: \"0af8544c-568c-4931-ba63-6aa8450fab29\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"
Sep 17 18:19:49.290730 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.290503    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0af8544c-568c-4931-ba63-6aa8450fab29-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd\" (UID: \"0af8544c-568c-4931-ba63-6aa8450fab29\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"
Sep 17 18:19:49.290730 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.290546    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0af8544c-568c-4931-ba63-6aa8450fab29-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd\" (UID: \"0af8544c-568c-4931-ba63-6aa8450fab29\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"
Sep 17 18:19:49.355217 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.355179    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs"]
Sep 17 18:19:49.358741 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.358718    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs"
Sep 17 18:19:49.360959 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.360939    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-lrqcc\""
Sep 17 18:19:49.372506 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.372483    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs"]
Sep 17 18:19:49.391545 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.391484    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0af8544c-568c-4931-ba63-6aa8450fab29-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd\" (UID: \"0af8544c-568c-4931-ba63-6aa8450fab29\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"
Sep 17 18:19:49.391545 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.391528    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2gdc\" (UniqueName: \"kubernetes.io/projected/0af8544c-568c-4931-ba63-6aa8450fab29-kube-api-access-x2gdc\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd\" (UID: \"0af8544c-568c-4931-ba63-6aa8450fab29\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"
Sep 17 18:19:49.391701 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.391557    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af8544c-568c-4931-ba63-6aa8450fab29-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd\" (UID: \"0af8544c-568c-4931-ba63-6aa8450fab29\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"
Sep 17 18:19:49.391701 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.391585    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af8544c-568c-4931-ba63-6aa8450fab29-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd\" (UID: \"0af8544c-568c-4931-ba63-6aa8450fab29\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"
Sep 17 18:19:49.391701 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.391637    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0af8544c-568c-4931-ba63-6aa8450fab29-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd\" (UID: \"0af8544c-568c-4931-ba63-6aa8450fab29\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"
Sep 17 18:19:49.391701 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.391691    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af8544c-568c-4931-ba63-6aa8450fab29-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd\" (UID: \"0af8544c-568c-4931-ba63-6aa8450fab29\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"
Sep 17 18:19:49.391916 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.391716    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af8544c-568c-4931-ba63-6aa8450fab29-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd\" (UID: \"0af8544c-568c-4931-ba63-6aa8450fab29\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"
Sep 17 18:19:49.391916 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.391747    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0af8544c-568c-4931-ba63-6aa8450fab29-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd\" (UID: \"0af8544c-568c-4931-ba63-6aa8450fab29\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"
Sep 17 18:19:49.392032 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.391955    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0af8544c-568c-4931-ba63-6aa8450fab29-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd\" (UID: \"0af8544c-568c-4931-ba63-6aa8450fab29\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"
Sep 17 18:19:49.392117 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.392097    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af8544c-568c-4931-ba63-6aa8450fab29-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd\" (UID: \"0af8544c-568c-4931-ba63-6aa8450fab29\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"
Sep 17 18:19:49.392182 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.392159    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0af8544c-568c-4931-ba63-6aa8450fab29-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd\" (UID: \"0af8544c-568c-4931-ba63-6aa8450fab29\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"
Sep 17 18:19:49.392232 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.392191    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af8544c-568c-4931-ba63-6aa8450fab29-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd\" (UID: \"0af8544c-568c-4931-ba63-6aa8450fab29\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"
Sep 17 18:19:49.392472 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.392451    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0af8544c-568c-4931-ba63-6aa8450fab29-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd\" (UID: \"0af8544c-568c-4931-ba63-6aa8450fab29\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"
Sep 17 18:19:49.393915 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.393891    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af8544c-568c-4931-ba63-6aa8450fab29-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd\" (UID: \"0af8544c-568c-4931-ba63-6aa8450fab29\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"
Sep 17 18:19:49.393997 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.393988    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af8544c-568c-4931-ba63-6aa8450fab29-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd\" (UID: \"0af8544c-568c-4931-ba63-6aa8450fab29\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"
Sep 17 18:19:49.403763 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.403742    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2gdc\" (UniqueName: \"kubernetes.io/projected/0af8544c-568c-4931-ba63-6aa8450fab29-kube-api-access-x2gdc\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd\" (UID: \"0af8544c-568c-4931-ba63-6aa8450fab29\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"
Sep 17 18:19:49.438749 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.438731    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"
Sep 17 18:19:49.493166 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.493128    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftjmg\" (UniqueName: \"kubernetes.io/projected/8d34bf9f-881f-4978-ac5c-f2cc672c4161-kube-api-access-ftjmg\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs\" (UID: \"8d34bf9f-881f-4978-ac5c-f2cc672c4161\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs"
Sep 17 18:19:49.493295 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.493226    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d34bf9f-881f-4978-ac5c-f2cc672c4161-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs\" (UID: \"8d34bf9f-881f-4978-ac5c-f2cc672c4161\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs"
Sep 17 18:19:49.594358 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.594319    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d34bf9f-881f-4978-ac5c-f2cc672c4161-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs\" (UID: \"8d34bf9f-881f-4978-ac5c-f2cc672c4161\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs"
Sep 17 18:19:49.594528 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.594386    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftjmg\" (UniqueName: \"kubernetes.io/projected/8d34bf9f-881f-4978-ac5c-f2cc672c4161-kube-api-access-ftjmg\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs\" (UID: \"8d34bf9f-881f-4978-ac5c-f2cc672c4161\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs"
Sep 17 18:19:49.596768 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.596750    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d34bf9f-881f-4978-ac5c-f2cc672c4161-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs\" (UID: \"8d34bf9f-881f-4978-ac5c-f2cc672c4161\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs"
Sep 17 18:19:49.602673 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.602639    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftjmg\" (UniqueName: \"kubernetes.io/projected/8d34bf9f-881f-4978-ac5c-f2cc672c4161-kube-api-access-ftjmg\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs\" (UID: \"8d34bf9f-881f-4978-ac5c-f2cc672c4161\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs"
Sep 17 18:19:49.670689 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.670606    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs"
Sep 17 18:19:49.722055 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.721979    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"]
Sep 17 18:19:49.881315 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.875885    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd" event={"ID":"0af8544c-568c-4931-ba63-6aa8450fab29","Type":"ContainerStarted","Data":"7874669265c46e1ae01e04decfcc3fbbaec03b2de6bca4cf62d48c6be23ab042"}
Sep 17 18:19:49.881315 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.876003    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd" event={"ID":"0af8544c-568c-4931-ba63-6aa8450fab29","Type":"ContainerStarted","Data":"0d2290a0f14ab78e42ad1664183542973d93d2c2f1ca0b47cca2c40b789ffebb"}
Sep 17 18:19:49.940982 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:49.940900    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs"]
Sep 17 18:19:49.944230 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:19:49.944200    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d34bf9f_881f_4978_ac5c_f2cc672c4161.slice/crio-8ff976803bca0c577dbe3600b303070170ec18223932aa502fbb157335f2a019 WatchSource:0}: Error finding container 8ff976803bca0c577dbe3600b303070170ec18223932aa502fbb157335f2a019: Status 404 returned error can't find the container with id 8ff976803bca0c577dbe3600b303070170ec18223932aa502fbb157335f2a019
Sep 17 18:19:50.884062 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:50.884023    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs" event={"ID":"8d34bf9f-881f-4978-ac5c-f2cc672c4161","Type":"ContainerStarted","Data":"e3ae37dd0203bc085e35dc46ad59c62cb55e3cf3e15f25aa186d016af2046585"}
Sep 17 18:19:50.884738 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:50.884649    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs" event={"ID":"8d34bf9f-881f-4978-ac5c-f2cc672c4161","Type":"ContainerStarted","Data":"8ff976803bca0c577dbe3600b303070170ec18223932aa502fbb157335f2a019"}
Sep 17 18:19:50.901401 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:50.901347    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs" podStartSLOduration=1.901329343 podStartE2EDuration="1.901329343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:19:50.900155959 +0000 UTC m=+2701.991462723" watchObservedRunningTime="2026-09-17 18:19:50.901329343 +0000 UTC m=+2701.992636119"
Sep 17 18:19:51.888630 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:51.888590    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs"
Sep 17 18:19:52.894460 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:52.894429    2583 generic.go:356] "Generic (PLEG): container finished" podID="0af8544c-568c-4931-ba63-6aa8450fab29" containerID="7874669265c46e1ae01e04decfcc3fbbaec03b2de6bca4cf62d48c6be23ab042" exitCode=0
Sep 17 18:19:52.895031 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:52.894498    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd" event={"ID":"0af8544c-568c-4931-ba63-6aa8450fab29","Type":"ContainerDied","Data":"7874669265c46e1ae01e04decfcc3fbbaec03b2de6bca4cf62d48c6be23ab042"}
Sep 17 18:19:53.901731 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:53.901693    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd" event={"ID":"0af8544c-568c-4931-ba63-6aa8450fab29","Type":"ContainerStarted","Data":"13850f5a1cb180b6da4b6ce8255784b989b1346fe60419c139d580dd6becb50b"}
Sep 17 18:19:53.927437 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:53.927371    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd" podStartSLOduration=2.173245043 podStartE2EDuration="4.927337594s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.754092551s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:19:53.924974935 +0000 UTC m=+2705.016281727" watchObservedRunningTime="2026-09-17 18:19:53.927337594 +0000 UTC m=+2705.018644360"
Sep 17 18:19:54.439606 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:54.439574    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"
Sep 17 18:19:59.439302 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:59.439261    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"
Sep 17 18:19:59.440939 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:19:59.440905    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd" podUID="0af8544c-568c-4931-ba63-6aa8450fab29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 17 18:20:09.439483 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:20:09.439438    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd" podUID="0af8544c-568c-4931-ba63-6aa8450fab29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 17 18:20:19.440040 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:20:19.439989    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd" podUID="0af8544c-568c-4931-ba63-6aa8450fab29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 17 18:20:22.896910 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:20:22.896880    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs"
Sep 17 18:20:29.439498 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:20:29.439451    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd" podUID="0af8544c-568c-4931-ba63-6aa8450fab29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 17 18:20:39.439591 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:20:39.439528    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd" podUID="0af8544c-568c-4931-ba63-6aa8450fab29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 17 18:20:49.440214 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:20:49.440163    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd" podUID="0af8544c-568c-4931-ba63-6aa8450fab29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 17 18:20:59.439793 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:20:59.439750    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd" podUID="0af8544c-568c-4931-ba63-6aa8450fab29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 17 18:21:09.449892 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:09.449804    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"
Sep 17 18:21:09.457596 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:09.457570    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"
Sep 17 18:21:36.867530 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:36.867486    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"]
Sep 17 18:21:36.869069 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:36.869044    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs"]
Sep 17 18:21:42.896757 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:42.896718    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs" podUID="8d34bf9f-881f-4978-ac5c-f2cc672c4161" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:21:51.887574 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:51.887509    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs" podUID="8d34bf9f-881f-4978-ac5c-f2cc672c4161" containerName="main" containerID="cri-o://e3ae37dd0203bc085e35dc46ad59c62cb55e3cf3e15f25aa186d016af2046585" gracePeriod=45
Sep 17 18:21:51.889629 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:51.889584    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd" podUID="0af8544c-568c-4931-ba63-6aa8450fab29" containerName="main" containerID="cri-o://13850f5a1cb180b6da4b6ce8255784b989b1346fe60419c139d580dd6becb50b" gracePeriod=45
Sep 17 18:21:52.430676 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:52.430626    2583 generic.go:356] "Generic (PLEG): container finished" podID="0af8544c-568c-4931-ba63-6aa8450fab29" containerID="13850f5a1cb180b6da4b6ce8255784b989b1346fe60419c139d580dd6becb50b" exitCode=0
Sep 17 18:21:52.430847 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:52.430699    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd" event={"ID":"0af8544c-568c-4931-ba63-6aa8450fab29","Type":"ContainerDied","Data":"13850f5a1cb180b6da4b6ce8255784b989b1346fe60419c139d580dd6becb50b"}
Sep 17 18:21:52.451009 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:52.450978    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd" podUID="0af8544c-568c-4931-ba63-6aa8450fab29" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 17 18:21:52.531053 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:52.531030    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"
Sep 17 18:21:52.663775 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:52.663734    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af8544c-568c-4931-ba63-6aa8450fab29-dshm\") pod \"0af8544c-568c-4931-ba63-6aa8450fab29\" (UID: \"0af8544c-568c-4931-ba63-6aa8450fab29\") "
Sep 17 18:21:52.663775 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:52.663781    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0af8544c-568c-4931-ba63-6aa8450fab29-kserve-provision-location\") pod \"0af8544c-568c-4931-ba63-6aa8450fab29\" (UID: \"0af8544c-568c-4931-ba63-6aa8450fab29\") "
Sep 17 18:21:52.664015 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:52.663808    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af8544c-568c-4931-ba63-6aa8450fab29-tmp-dir\") pod \"0af8544c-568c-4931-ba63-6aa8450fab29\" (UID: \"0af8544c-568c-4931-ba63-6aa8450fab29\") "
Sep 17 18:21:52.664015 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:52.663848    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2gdc\" (UniqueName: \"kubernetes.io/projected/0af8544c-568c-4931-ba63-6aa8450fab29-kube-api-access-x2gdc\") pod \"0af8544c-568c-4931-ba63-6aa8450fab29\" (UID: \"0af8544c-568c-4931-ba63-6aa8450fab29\") "
Sep 17 18:21:52.664015 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:52.663875    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af8544c-568c-4931-ba63-6aa8450fab29-tls-certs\") pod \"0af8544c-568c-4931-ba63-6aa8450fab29\" (UID: \"0af8544c-568c-4931-ba63-6aa8450fab29\") "
Sep 17 18:21:52.664015 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:52.663891    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0af8544c-568c-4931-ba63-6aa8450fab29-home\") pod \"0af8544c-568c-4931-ba63-6aa8450fab29\" (UID: \"0af8544c-568c-4931-ba63-6aa8450fab29\") "
Sep 17 18:21:52.664015 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:52.663908    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af8544c-568c-4931-ba63-6aa8450fab29-model-cache\") pod \"0af8544c-568c-4931-ba63-6aa8450fab29\" (UID: \"0af8544c-568c-4931-ba63-6aa8450fab29\") "
Sep 17 18:21:52.664015 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:52.663926    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0af8544c-568c-4931-ba63-6aa8450fab29-cabundle-cert\") pod \"0af8544c-568c-4931-ba63-6aa8450fab29\" (UID: \"0af8544c-568c-4931-ba63-6aa8450fab29\") "
Sep 17 18:21:52.664416 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:52.664390    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af8544c-568c-4931-ba63-6aa8450fab29-home" (OuterVolumeSpecName: "home") pod "0af8544c-568c-4931-ba63-6aa8450fab29" (UID: "0af8544c-568c-4931-ba63-6aa8450fab29"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:21:52.664493 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:52.664417    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0af8544c-568c-4931-ba63-6aa8450fab29-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0af8544c-568c-4931-ba63-6aa8450fab29" (UID: "0af8544c-568c-4931-ba63-6aa8450fab29"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:21:52.664687 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:52.664633    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af8544c-568c-4931-ba63-6aa8450fab29-model-cache" (OuterVolumeSpecName: "model-cache") pod "0af8544c-568c-4931-ba63-6aa8450fab29" (UID: "0af8544c-568c-4931-ba63-6aa8450fab29"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:21:52.666182 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:52.666158    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0af8544c-568c-4931-ba63-6aa8450fab29-kube-api-access-x2gdc" (OuterVolumeSpecName: "kube-api-access-x2gdc") pod "0af8544c-568c-4931-ba63-6aa8450fab29" (UID: "0af8544c-568c-4931-ba63-6aa8450fab29"). InnerVolumeSpecName "kube-api-access-x2gdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:21:52.666360 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:52.666340    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af8544c-568c-4931-ba63-6aa8450fab29-dshm" (OuterVolumeSpecName: "dshm") pod "0af8544c-568c-4931-ba63-6aa8450fab29" (UID: "0af8544c-568c-4931-ba63-6aa8450fab29"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:21:52.666429 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:52.666410    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0af8544c-568c-4931-ba63-6aa8450fab29-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0af8544c-568c-4931-ba63-6aa8450fab29" (UID: "0af8544c-568c-4931-ba63-6aa8450fab29"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:21:52.681986 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:52.681952    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af8544c-568c-4931-ba63-6aa8450fab29-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0af8544c-568c-4931-ba63-6aa8450fab29" (UID: "0af8544c-568c-4931-ba63-6aa8450fab29"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:21:52.693089 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:52.693066    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af8544c-568c-4931-ba63-6aa8450fab29-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0af8544c-568c-4931-ba63-6aa8450fab29" (UID: "0af8544c-568c-4931-ba63-6aa8450fab29"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:21:52.765372 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:52.765350    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0af8544c-568c-4931-ba63-6aa8450fab29-dshm\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:21:52.765372 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:52.765373    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0af8544c-568c-4931-ba63-6aa8450fab29-kserve-provision-location\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:21:52.765509 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:52.765401    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0af8544c-568c-4931-ba63-6aa8450fab29-tmp-dir\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:21:52.765509 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:52.765410    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2gdc\" (UniqueName: \"kubernetes.io/projected/0af8544c-568c-4931-ba63-6aa8450fab29-kube-api-access-x2gdc\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:21:52.765509 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:52.765418    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0af8544c-568c-4931-ba63-6aa8450fab29-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:21:52.765509 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:52.765426    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0af8544c-568c-4931-ba63-6aa8450fab29-home\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:21:52.765509 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:52.765433    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0af8544c-568c-4931-ba63-6aa8450fab29-model-cache\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:21:52.765509 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:52.765441    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0af8544c-568c-4931-ba63-6aa8450fab29-cabundle-cert\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:21:52.896440 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:52.896414    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs" podUID="8d34bf9f-881f-4978-ac5c-f2cc672c4161" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:21:53.436233 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:53.436196    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd" event={"ID":"0af8544c-568c-4931-ba63-6aa8450fab29","Type":"ContainerDied","Data":"0d2290a0f14ab78e42ad1664183542973d93d2c2f1ca0b47cca2c40b789ffebb"}
Sep 17 18:21:53.436444 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:53.436263    2583 scope.go:117] "RemoveContainer" containerID="13850f5a1cb180b6da4b6ce8255784b989b1346fe60419c139d580dd6becb50b"
Sep 17 18:21:53.436444 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:53.436214    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"
Sep 17 18:21:53.444348 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:53.444333    2583 scope.go:117] "RemoveContainer" containerID="7874669265c46e1ae01e04decfcc3fbbaec03b2de6bca4cf62d48c6be23ab042"
Sep 17 18:21:53.459176 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:53.459153    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"]
Sep 17 18:21:53.463034 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:53.463012    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7d6bb7d67c-74bmd"]
Sep 17 18:21:55.040018 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:21:55.039986    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0af8544c-568c-4931-ba63-6aa8450fab29" path="/var/lib/kubelet/pods/0af8544c-568c-4931-ba63-6aa8450fab29/volumes"
Sep 17 18:22:02.896672 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:02.896633    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs" podUID="8d34bf9f-881f-4978-ac5c-f2cc672c4161" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:22:02.897071 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:02.896748    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs"
Sep 17 18:22:12.896120 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:12.896084    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs" podUID="8d34bf9f-881f-4978-ac5c-f2cc672c4161" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:22:22.120759 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:22.120734    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs"
Sep 17 18:22:22.191631 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:22.191561    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d34bf9f-881f-4978-ac5c-f2cc672c4161-tls-certs\") pod \"8d34bf9f-881f-4978-ac5c-f2cc672c4161\" (UID: \"8d34bf9f-881f-4978-ac5c-f2cc672c4161\") "
Sep 17 18:22:22.191812 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:22.191632    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftjmg\" (UniqueName: \"kubernetes.io/projected/8d34bf9f-881f-4978-ac5c-f2cc672c4161-kube-api-access-ftjmg\") pod \"8d34bf9f-881f-4978-ac5c-f2cc672c4161\" (UID: \"8d34bf9f-881f-4978-ac5c-f2cc672c4161\") "
Sep 17 18:22:22.193758 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:22.193731    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d34bf9f-881f-4978-ac5c-f2cc672c4161-kube-api-access-ftjmg" (OuterVolumeSpecName: "kube-api-access-ftjmg") pod "8d34bf9f-881f-4978-ac5c-f2cc672c4161" (UID: "8d34bf9f-881f-4978-ac5c-f2cc672c4161"). InnerVolumeSpecName "kube-api-access-ftjmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:22:22.193858 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:22.193738    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d34bf9f-881f-4978-ac5c-f2cc672c4161-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8d34bf9f-881f-4978-ac5c-f2cc672c4161" (UID: "8d34bf9f-881f-4978-ac5c-f2cc672c4161"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:22:22.292798 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:22.292770    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d34bf9f-881f-4978-ac5c-f2cc672c4161-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:22:22.292798 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:22.292795    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftjmg\" (UniqueName: \"kubernetes.io/projected/8d34bf9f-881f-4978-ac5c-f2cc672c4161-kube-api-access-ftjmg\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:22:22.562749 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:22.562640    2583 generic.go:356] "Generic (PLEG): container finished" podID="8d34bf9f-881f-4978-ac5c-f2cc672c4161" containerID="e3ae37dd0203bc085e35dc46ad59c62cb55e3cf3e15f25aa186d016af2046585" exitCode=0
Sep 17 18:22:22.562749 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:22.562697    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs"
Sep 17 18:22:22.562749 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:22.562726    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs" event={"ID":"8d34bf9f-881f-4978-ac5c-f2cc672c4161","Type":"ContainerDied","Data":"e3ae37dd0203bc085e35dc46ad59c62cb55e3cf3e15f25aa186d016af2046585"}
Sep 17 18:22:22.563048 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:22.562790    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs" event={"ID":"8d34bf9f-881f-4978-ac5c-f2cc672c4161","Type":"ContainerDied","Data":"8ff976803bca0c577dbe3600b303070170ec18223932aa502fbb157335f2a019"}
Sep 17 18:22:22.563048 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:22.562819    2583 scope.go:117] "RemoveContainer" containerID="e3ae37dd0203bc085e35dc46ad59c62cb55e3cf3e15f25aa186d016af2046585"
Sep 17 18:22:22.571105 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:22.571087    2583 scope.go:117] "RemoveContainer" containerID="e3ae37dd0203bc085e35dc46ad59c62cb55e3cf3e15f25aa186d016af2046585"
Sep 17 18:22:22.571372 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:22:22.571355    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3ae37dd0203bc085e35dc46ad59c62cb55e3cf3e15f25aa186d016af2046585\": container with ID starting with e3ae37dd0203bc085e35dc46ad59c62cb55e3cf3e15f25aa186d016af2046585 not found: ID does not exist" containerID="e3ae37dd0203bc085e35dc46ad59c62cb55e3cf3e15f25aa186d016af2046585"
Sep 17 18:22:22.571416 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:22.571380    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3ae37dd0203bc085e35dc46ad59c62cb55e3cf3e15f25aa186d016af2046585"} err="failed to get container status \"e3ae37dd0203bc085e35dc46ad59c62cb55e3cf3e15f25aa186d016af2046585\": rpc error: code = NotFound desc = could not find container \"e3ae37dd0203bc085e35dc46ad59c62cb55e3cf3e15f25aa186d016af2046585\": container with ID starting with e3ae37dd0203bc085e35dc46ad59c62cb55e3cf3e15f25aa186d016af2046585 not found: ID does not exist"
Sep 17 18:22:22.585814 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:22.585792    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs"]
Sep 17 18:22:22.593000 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:22.592980    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scher8xrs"]
Sep 17 18:22:23.039689 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:23.039646    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d34bf9f-881f-4978-ac5c-f2cc672c4161" path="/var/lib/kubelet/pods/8d34bf9f-881f-4978-ac5c-f2cc672c4161/volumes"
Sep 17 18:22:35.426282 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.426246    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"]
Sep 17 18:22:35.426765 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.426575    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0af8544c-568c-4931-ba63-6aa8450fab29" containerName="main"
Sep 17 18:22:35.426765 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.426586    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0af8544c-568c-4931-ba63-6aa8450fab29" containerName="main"
Sep 17 18:22:35.426765 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.426596    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0af8544c-568c-4931-ba63-6aa8450fab29" containerName="storage-initializer"
Sep 17 18:22:35.426765 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.426602    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0af8544c-568c-4931-ba63-6aa8450fab29" containerName="storage-initializer"
Sep 17 18:22:35.426765 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.426610    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d34bf9f-881f-4978-ac5c-f2cc672c4161" containerName="main"
Sep 17 18:22:35.426765 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.426615    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d34bf9f-881f-4978-ac5c-f2cc672c4161" containerName="main"
Sep 17 18:22:35.426765 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.426693    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d34bf9f-881f-4978-ac5c-f2cc672c4161" containerName="main"
Sep 17 18:22:35.426765 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.426701    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0af8544c-568c-4931-ba63-6aa8450fab29" containerName="main"
Sep 17 18:22:35.429236 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.429218    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"
Sep 17 18:22:35.431991 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.431965    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 17 18:22:35.432118 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.431972    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:22:35.432118 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.432049    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Sep 17 18:22:35.432118 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.432073    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 17 18:22:35.432270 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.432123    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-qn7pg\""
Sep 17 18:22:35.432270 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.432132    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 17 18:22:35.444307 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.444284    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"]
Sep 17 18:22:35.594920 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.594893    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/919602a1-7927-42a4-ad63-080b0bb3d0f6-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9\" (UID: \"919602a1-7927-42a4-ad63-080b0bb3d0f6\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"
Sep 17 18:22:35.595103 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.594927    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/919602a1-7927-42a4-ad63-080b0bb3d0f6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9\" (UID: \"919602a1-7927-42a4-ad63-080b0bb3d0f6\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"
Sep 17 18:22:35.595103 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.594952    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/919602a1-7927-42a4-ad63-080b0bb3d0f6-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9\" (UID: \"919602a1-7927-42a4-ad63-080b0bb3d0f6\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"
Sep 17 18:22:35.595103 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.595039    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/919602a1-7927-42a4-ad63-080b0bb3d0f6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9\" (UID: \"919602a1-7927-42a4-ad63-080b0bb3d0f6\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"
Sep 17 18:22:35.595103 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.595081    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvpms\" (UniqueName: \"kubernetes.io/projected/919602a1-7927-42a4-ad63-080b0bb3d0f6-kube-api-access-jvpms\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9\" (UID: \"919602a1-7927-42a4-ad63-080b0bb3d0f6\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"
Sep 17 18:22:35.595103 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.595098    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/919602a1-7927-42a4-ad63-080b0bb3d0f6-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9\" (UID: \"919602a1-7927-42a4-ad63-080b0bb3d0f6\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"
Sep 17 18:22:35.595318 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.595137    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/919602a1-7927-42a4-ad63-080b0bb3d0f6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9\" (UID: \"919602a1-7927-42a4-ad63-080b0bb3d0f6\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"
Sep 17 18:22:35.595318 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.595154    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/919602a1-7927-42a4-ad63-080b0bb3d0f6-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9\" (UID: \"919602a1-7927-42a4-ad63-080b0bb3d0f6\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"
Sep 17 18:22:35.695511 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.695432    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/919602a1-7927-42a4-ad63-080b0bb3d0f6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9\" (UID: \"919602a1-7927-42a4-ad63-080b0bb3d0f6\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"
Sep 17 18:22:35.695511 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.695469    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvpms\" (UniqueName: \"kubernetes.io/projected/919602a1-7927-42a4-ad63-080b0bb3d0f6-kube-api-access-jvpms\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9\" (UID: \"919602a1-7927-42a4-ad63-080b0bb3d0f6\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"
Sep 17 18:22:35.695511 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.695487    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/919602a1-7927-42a4-ad63-080b0bb3d0f6-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9\" (UID: \"919602a1-7927-42a4-ad63-080b0bb3d0f6\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"
Sep 17 18:22:35.695821 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.695526    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/919602a1-7927-42a4-ad63-080b0bb3d0f6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9\" (UID: \"919602a1-7927-42a4-ad63-080b0bb3d0f6\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"
Sep 17 18:22:35.695821 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.695544    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/919602a1-7927-42a4-ad63-080b0bb3d0f6-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9\" (UID: \"919602a1-7927-42a4-ad63-080b0bb3d0f6\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"
Sep 17 18:22:35.695821 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.695558    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/919602a1-7927-42a4-ad63-080b0bb3d0f6-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9\" (UID: \"919602a1-7927-42a4-ad63-080b0bb3d0f6\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"
Sep 17 18:22:35.695821 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.695580    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/919602a1-7927-42a4-ad63-080b0bb3d0f6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9\" (UID: \"919602a1-7927-42a4-ad63-080b0bb3d0f6\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"
Sep 17 18:22:35.695821 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.695601    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/919602a1-7927-42a4-ad63-080b0bb3d0f6-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9\" (UID: \"919602a1-7927-42a4-ad63-080b0bb3d0f6\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"
Sep 17 18:22:35.696072 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.696052    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/919602a1-7927-42a4-ad63-080b0bb3d0f6-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9\" (UID: \"919602a1-7927-42a4-ad63-080b0bb3d0f6\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"
Sep 17 18:22:35.696142 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.696119    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/919602a1-7927-42a4-ad63-080b0bb3d0f6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9\" (UID: \"919602a1-7927-42a4-ad63-080b0bb3d0f6\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"
Sep 17 18:22:35.696219 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.696194    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/919602a1-7927-42a4-ad63-080b0bb3d0f6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9\" (UID: \"919602a1-7927-42a4-ad63-080b0bb3d0f6\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"
Sep 17 18:22:35.696270 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.696249    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/919602a1-7927-42a4-ad63-080b0bb3d0f6-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9\" (UID: \"919602a1-7927-42a4-ad63-080b0bb3d0f6\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"
Sep 17 18:22:35.696433 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.696415    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/919602a1-7927-42a4-ad63-080b0bb3d0f6-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9\" (UID: \"919602a1-7927-42a4-ad63-080b0bb3d0f6\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"
Sep 17 18:22:35.697886 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.697863    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/919602a1-7927-42a4-ad63-080b0bb3d0f6-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9\" (UID: \"919602a1-7927-42a4-ad63-080b0bb3d0f6\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"
Sep 17 18:22:35.698134 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.698116    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/919602a1-7927-42a4-ad63-080b0bb3d0f6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9\" (UID: \"919602a1-7927-42a4-ad63-080b0bb3d0f6\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"
Sep 17 18:22:35.713855 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.713831    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvpms\" (UniqueName: \"kubernetes.io/projected/919602a1-7927-42a4-ad63-080b0bb3d0f6-kube-api-access-jvpms\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9\" (UID: \"919602a1-7927-42a4-ad63-080b0bb3d0f6\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"
Sep 17 18:22:35.739629 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:35.739604    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"
Sep 17 18:22:36.017852 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:36.017825    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"]
Sep 17 18:22:36.019598 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:22:36.019574    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod919602a1_7927_42a4_ad63_080b0bb3d0f6.slice/crio-516e8526aa59427d88e5f7e35f22fadfcbed814c7d02af4cd603f3eebe319842 WatchSource:0}: Error finding container 516e8526aa59427d88e5f7e35f22fadfcbed814c7d02af4cd603f3eebe319842: Status 404 returned error can't find the container with id 516e8526aa59427d88e5f7e35f22fadfcbed814c7d02af4cd603f3eebe319842
Sep 17 18:22:36.624615 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:36.624579    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9" event={"ID":"919602a1-7927-42a4-ad63-080b0bb3d0f6","Type":"ContainerStarted","Data":"3b75fbf5bed90aa9d133f0368f63071197b337ca1cd60fa6062f80773b4b558c"}
Sep 17 18:22:36.625036 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:36.624633    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9" event={"ID":"919602a1-7927-42a4-ad63-080b0bb3d0f6","Type":"ContainerStarted","Data":"516e8526aa59427d88e5f7e35f22fadfcbed814c7d02af4cd603f3eebe319842"}
Sep 17 18:22:39.638948 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:39.638914    2583 generic.go:356] "Generic (PLEG): container finished" podID="919602a1-7927-42a4-ad63-080b0bb3d0f6" containerID="3b75fbf5bed90aa9d133f0368f63071197b337ca1cd60fa6062f80773b4b558c" exitCode=0
Sep 17 18:22:39.638948 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:39.638951    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9" event={"ID":"919602a1-7927-42a4-ad63-080b0bb3d0f6","Type":"ContainerDied","Data":"3b75fbf5bed90aa9d133f0368f63071197b337ca1cd60fa6062f80773b4b558c"}
Sep 17 18:22:40.645853 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:40.645820    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9" event={"ID":"919602a1-7927-42a4-ad63-080b0bb3d0f6","Type":"ContainerStarted","Data":"238dfed7ed007874f6b0ea2d312f6e466a0bbee1b4112ad76cb0249666f6b5ee"}
Sep 17 18:22:40.669711 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:40.669642    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9" podStartSLOduration=2.995430054 podStartE2EDuration="5.669622653s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.674192599s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:22:40.668651667 +0000 UTC m=+2871.759958432" watchObservedRunningTime="2026-09-17 18:22:40.669622653 +0000 UTC m=+2871.760929419"
Sep 17 18:22:40.739770 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:40.739732    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"
Sep 17 18:22:45.740680 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:45.740631    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"
Sep 17 18:22:45.742129 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:45.742106    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9" podUID="919602a1-7927-42a4-ad63-080b0bb3d0f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 17 18:22:55.740328 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:22:55.740268    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9" podUID="919602a1-7927-42a4-ad63-080b0bb3d0f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 17 18:23:05.740640 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:23:05.740597    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9" podUID="919602a1-7927-42a4-ad63-080b0bb3d0f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 17 18:23:15.740692 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:23:15.740632    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9" podUID="919602a1-7927-42a4-ad63-080b0bb3d0f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 17 18:23:25.740649 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:23:25.740603    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9" podUID="919602a1-7927-42a4-ad63-080b0bb3d0f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 17 18:23:35.741013 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:23:35.740971    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9" podUID="919602a1-7927-42a4-ad63-080b0bb3d0f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 17 18:23:45.739995 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:23:45.739953    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9" podUID="919602a1-7927-42a4-ad63-080b0bb3d0f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 17 18:23:55.749688 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:23:55.749643    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"
Sep 17 18:23:55.992371 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:23:55.992337    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"
Sep 17 18:23:57.656906 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:23:57.656874    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"]
Sep 17 18:24:13.014713 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:13.014652    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9" podUID="919602a1-7927-42a4-ad63-080b0bb3d0f6" containerName="main" containerID="cri-o://238dfed7ed007874f6b0ea2d312f6e466a0bbee1b4112ad76cb0249666f6b5ee" gracePeriod=45
Sep 17 18:24:13.559181 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:13.559150    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"
Sep 17 18:24:13.697209 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:13.697122    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/919602a1-7927-42a4-ad63-080b0bb3d0f6-model-cache\") pod \"919602a1-7927-42a4-ad63-080b0bb3d0f6\" (UID: \"919602a1-7927-42a4-ad63-080b0bb3d0f6\") "
Sep 17 18:24:13.697209 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:13.697158    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/919602a1-7927-42a4-ad63-080b0bb3d0f6-dshm\") pod \"919602a1-7927-42a4-ad63-080b0bb3d0f6\" (UID: \"919602a1-7927-42a4-ad63-080b0bb3d0f6\") "
Sep 17 18:24:13.697209 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:13.697185    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/919602a1-7927-42a4-ad63-080b0bb3d0f6-home\") pod \"919602a1-7927-42a4-ad63-080b0bb3d0f6\" (UID: \"919602a1-7927-42a4-ad63-080b0bb3d0f6\") "
Sep 17 18:24:13.697209 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:13.697207    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/919602a1-7927-42a4-ad63-080b0bb3d0f6-tls-certs\") pod \"919602a1-7927-42a4-ad63-080b0bb3d0f6\" (UID: \"919602a1-7927-42a4-ad63-080b0bb3d0f6\") "
Sep 17 18:24:13.697549 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:13.697240    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/919602a1-7927-42a4-ad63-080b0bb3d0f6-tmp-dir\") pod \"919602a1-7927-42a4-ad63-080b0bb3d0f6\" (UID: \"919602a1-7927-42a4-ad63-080b0bb3d0f6\") "
Sep 17 18:24:13.697549 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:13.697309    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvpms\" (UniqueName: \"kubernetes.io/projected/919602a1-7927-42a4-ad63-080b0bb3d0f6-kube-api-access-jvpms\") pod \"919602a1-7927-42a4-ad63-080b0bb3d0f6\" (UID: \"919602a1-7927-42a4-ad63-080b0bb3d0f6\") "
Sep 17 18:24:13.697549 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:13.697336    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/919602a1-7927-42a4-ad63-080b0bb3d0f6-cabundle-cert\") pod \"919602a1-7927-42a4-ad63-080b0bb3d0f6\" (UID: \"919602a1-7927-42a4-ad63-080b0bb3d0f6\") "
Sep 17 18:24:13.697549 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:13.697378    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/919602a1-7927-42a4-ad63-080b0bb3d0f6-kserve-provision-location\") pod \"919602a1-7927-42a4-ad63-080b0bb3d0f6\" (UID: \"919602a1-7927-42a4-ad63-080b0bb3d0f6\") "
Sep 17 18:24:13.697549 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:13.697406    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919602a1-7927-42a4-ad63-080b0bb3d0f6-model-cache" (OuterVolumeSpecName: "model-cache") pod "919602a1-7927-42a4-ad63-080b0bb3d0f6" (UID: "919602a1-7927-42a4-ad63-080b0bb3d0f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:24:13.697851 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:13.697630    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/919602a1-7927-42a4-ad63-080b0bb3d0f6-model-cache\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:24:13.697851 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:13.697570    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919602a1-7927-42a4-ad63-080b0bb3d0f6-home" (OuterVolumeSpecName: "home") pod "919602a1-7927-42a4-ad63-080b0bb3d0f6" (UID: "919602a1-7927-42a4-ad63-080b0bb3d0f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:24:13.697851 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:13.697765    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919602a1-7927-42a4-ad63-080b0bb3d0f6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "919602a1-7927-42a4-ad63-080b0bb3d0f6" (UID: "919602a1-7927-42a4-ad63-080b0bb3d0f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:24:13.698021 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:13.697935    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/919602a1-7927-42a4-ad63-080b0bb3d0f6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "919602a1-7927-42a4-ad63-080b0bb3d0f6" (UID: "919602a1-7927-42a4-ad63-080b0bb3d0f6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:24:13.699628 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:13.699603    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919602a1-7927-42a4-ad63-080b0bb3d0f6-dshm" (OuterVolumeSpecName: "dshm") pod "919602a1-7927-42a4-ad63-080b0bb3d0f6" (UID: "919602a1-7927-42a4-ad63-080b0bb3d0f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:24:13.699790 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:13.699700    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/919602a1-7927-42a4-ad63-080b0bb3d0f6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "919602a1-7927-42a4-ad63-080b0bb3d0f6" (UID: "919602a1-7927-42a4-ad63-080b0bb3d0f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:24:13.699790 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:13.699758    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919602a1-7927-42a4-ad63-080b0bb3d0f6-kube-api-access-jvpms" (OuterVolumeSpecName: "kube-api-access-jvpms") pod "919602a1-7927-42a4-ad63-080b0bb3d0f6" (UID: "919602a1-7927-42a4-ad63-080b0bb3d0f6"). InnerVolumeSpecName "kube-api-access-jvpms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:24:13.720721 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:13.720693    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919602a1-7927-42a4-ad63-080b0bb3d0f6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "919602a1-7927-42a4-ad63-080b0bb3d0f6" (UID: "919602a1-7927-42a4-ad63-080b0bb3d0f6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:24:13.798897 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:13.798870    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/919602a1-7927-42a4-ad63-080b0bb3d0f6-cabundle-cert\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:24:13.798897 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:13.798897    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/919602a1-7927-42a4-ad63-080b0bb3d0f6-kserve-provision-location\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:24:13.799065 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:13.798911    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/919602a1-7927-42a4-ad63-080b0bb3d0f6-dshm\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:24:13.799065 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:13.798924    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/919602a1-7927-42a4-ad63-080b0bb3d0f6-home\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:24:13.799065 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:13.798934    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/919602a1-7927-42a4-ad63-080b0bb3d0f6-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:24:13.799065 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:13.798969    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/919602a1-7927-42a4-ad63-080b0bb3d0f6-tmp-dir\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:24:13.799065 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:13.798982    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvpms\" (UniqueName: \"kubernetes.io/projected/919602a1-7927-42a4-ad63-080b0bb3d0f6-kube-api-access-jvpms\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:24:14.069199 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:14.069111    2583 generic.go:356] "Generic (PLEG): container finished" podID="919602a1-7927-42a4-ad63-080b0bb3d0f6" containerID="238dfed7ed007874f6b0ea2d312f6e466a0bbee1b4112ad76cb0249666f6b5ee" exitCode=0
Sep 17 18:24:14.069199 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:14.069157    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"
Sep 17 18:24:14.069842 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:14.069195    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9" event={"ID":"919602a1-7927-42a4-ad63-080b0bb3d0f6","Type":"ContainerDied","Data":"238dfed7ed007874f6b0ea2d312f6e466a0bbee1b4112ad76cb0249666f6b5ee"}
Sep 17 18:24:14.069842 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:14.069248    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9" event={"ID":"919602a1-7927-42a4-ad63-080b0bb3d0f6","Type":"ContainerDied","Data":"516e8526aa59427d88e5f7e35f22fadfcbed814c7d02af4cd603f3eebe319842"}
Sep 17 18:24:14.069842 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:14.069271    2583 scope.go:117] "RemoveContainer" containerID="238dfed7ed007874f6b0ea2d312f6e466a0bbee1b4112ad76cb0249666f6b5ee"
Sep 17 18:24:14.080480 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:14.080459    2583 scope.go:117] "RemoveContainer" containerID="3b75fbf5bed90aa9d133f0368f63071197b337ca1cd60fa6062f80773b4b558c"
Sep 17 18:24:14.088216 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:14.088193    2583 scope.go:117] "RemoveContainer" containerID="238dfed7ed007874f6b0ea2d312f6e466a0bbee1b4112ad76cb0249666f6b5ee"
Sep 17 18:24:14.088465 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:24:14.088446    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"238dfed7ed007874f6b0ea2d312f6e466a0bbee1b4112ad76cb0249666f6b5ee\": container with ID starting with 238dfed7ed007874f6b0ea2d312f6e466a0bbee1b4112ad76cb0249666f6b5ee not found: ID does not exist" containerID="238dfed7ed007874f6b0ea2d312f6e466a0bbee1b4112ad76cb0249666f6b5ee"
Sep 17 18:24:14.088525 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:14.088473    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"238dfed7ed007874f6b0ea2d312f6e466a0bbee1b4112ad76cb0249666f6b5ee"} err="failed to get container status \"238dfed7ed007874f6b0ea2d312f6e466a0bbee1b4112ad76cb0249666f6b5ee\": rpc error: code = NotFound desc = could not find container \"238dfed7ed007874f6b0ea2d312f6e466a0bbee1b4112ad76cb0249666f6b5ee\": container with ID starting with 238dfed7ed007874f6b0ea2d312f6e466a0bbee1b4112ad76cb0249666f6b5ee not found: ID does not exist"
Sep 17 18:24:14.088525 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:14.088490    2583 scope.go:117] "RemoveContainer" containerID="3b75fbf5bed90aa9d133f0368f63071197b337ca1cd60fa6062f80773b4b558c"
Sep 17 18:24:14.088754 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:24:14.088725    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b75fbf5bed90aa9d133f0368f63071197b337ca1cd60fa6062f80773b4b558c\": container with ID starting with 3b75fbf5bed90aa9d133f0368f63071197b337ca1cd60fa6062f80773b4b558c not found: ID does not exist" containerID="3b75fbf5bed90aa9d133f0368f63071197b337ca1cd60fa6062f80773b4b558c"
Sep 17 18:24:14.088836 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:14.088763    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b75fbf5bed90aa9d133f0368f63071197b337ca1cd60fa6062f80773b4b558c"} err="failed to get container status \"3b75fbf5bed90aa9d133f0368f63071197b337ca1cd60fa6062f80773b4b558c\": rpc error: code = NotFound desc = could not find container \"3b75fbf5bed90aa9d133f0368f63071197b337ca1cd60fa6062f80773b4b558c\": container with ID starting with 3b75fbf5bed90aa9d133f0368f63071197b337ca1cd60fa6062f80773b4b558c not found: ID does not exist"
Sep 17 18:24:14.093309 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:14.093280    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"]
Sep 17 18:24:14.097355 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:14.097336    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7f596fbd6b-qqzk9"]
Sep 17 18:24:15.040336 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:15.040302    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="919602a1-7927-42a4-ad63-080b0bb3d0f6" path="/var/lib/kubelet/pods/919602a1-7927-42a4-ad63-080b0bb3d0f6/volumes"
Sep 17 18:24:19.322413 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.322374    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"]
Sep 17 18:24:19.322908 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.322825    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="919602a1-7927-42a4-ad63-080b0bb3d0f6" containerName="storage-initializer"
Sep 17 18:24:19.322908 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.322844    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="919602a1-7927-42a4-ad63-080b0bb3d0f6" containerName="storage-initializer"
Sep 17 18:24:19.322908 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.322876    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="919602a1-7927-42a4-ad63-080b0bb3d0f6" containerName="main"
Sep 17 18:24:19.322908 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.322884    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="919602a1-7927-42a4-ad63-080b0bb3d0f6" containerName="main"
Sep 17 18:24:19.323182 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.322956    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="919602a1-7927-42a4-ad63-080b0bb3d0f6" containerName="main"
Sep 17 18:24:19.327564 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.327539    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"
Sep 17 18:24:19.330044 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.330018    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 17 18:24:19.330044 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.330036    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Sep 17 18:24:19.330239 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.330018    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:24:19.330375 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.330357    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 17 18:24:19.330422 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.330402    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 17 18:24:19.330472 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.330454    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-l9zxl\""
Sep 17 18:24:19.337990 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.337970    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"]
Sep 17 18:24:19.441485 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.441456    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58dqj\" (UniqueName: \"kubernetes.io/projected/a3a55ec7-6060-4d7a-ad52-1696523129f9-kube-api-access-58dqj\") pod \"router-with-refs-test-kserve-76f47bf4f8-qtvn6\" (UID: \"a3a55ec7-6060-4d7a-ad52-1696523129f9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"
Sep 17 18:24:19.441649 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.441490    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3a55ec7-6060-4d7a-ad52-1696523129f9-dshm\") pod \"router-with-refs-test-kserve-76f47bf4f8-qtvn6\" (UID: \"a3a55ec7-6060-4d7a-ad52-1696523129f9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"
Sep 17 18:24:19.441649 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.441513    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3a55ec7-6060-4d7a-ad52-1696523129f9-tls-certs\") pod \"router-with-refs-test-kserve-76f47bf4f8-qtvn6\" (UID: \"a3a55ec7-6060-4d7a-ad52-1696523129f9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"
Sep 17 18:24:19.441649 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.441529    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3a55ec7-6060-4d7a-ad52-1696523129f9-kserve-provision-location\") pod \"router-with-refs-test-kserve-76f47bf4f8-qtvn6\" (UID: \"a3a55ec7-6060-4d7a-ad52-1696523129f9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"
Sep 17 18:24:19.441649 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.441574    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3a55ec7-6060-4d7a-ad52-1696523129f9-cabundle-cert\") pod \"router-with-refs-test-kserve-76f47bf4f8-qtvn6\" (UID: \"a3a55ec7-6060-4d7a-ad52-1696523129f9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"
Sep 17 18:24:19.441877 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.441696    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3a55ec7-6060-4d7a-ad52-1696523129f9-home\") pod \"router-with-refs-test-kserve-76f47bf4f8-qtvn6\" (UID: \"a3a55ec7-6060-4d7a-ad52-1696523129f9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"
Sep 17 18:24:19.441877 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.441726    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3a55ec7-6060-4d7a-ad52-1696523129f9-model-cache\") pod \"router-with-refs-test-kserve-76f47bf4f8-qtvn6\" (UID: \"a3a55ec7-6060-4d7a-ad52-1696523129f9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"
Sep 17 18:24:19.441877 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.441768    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3a55ec7-6060-4d7a-ad52-1696523129f9-tmp-dir\") pod \"router-with-refs-test-kserve-76f47bf4f8-qtvn6\" (UID: \"a3a55ec7-6060-4d7a-ad52-1696523129f9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"
Sep 17 18:24:19.543000 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.542972    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3a55ec7-6060-4d7a-ad52-1696523129f9-home\") pod \"router-with-refs-test-kserve-76f47bf4f8-qtvn6\" (UID: \"a3a55ec7-6060-4d7a-ad52-1696523129f9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"
Sep 17 18:24:19.543173 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.543013    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3a55ec7-6060-4d7a-ad52-1696523129f9-model-cache\") pod \"router-with-refs-test-kserve-76f47bf4f8-qtvn6\" (UID: \"a3a55ec7-6060-4d7a-ad52-1696523129f9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"
Sep 17 18:24:19.543173 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.543053    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3a55ec7-6060-4d7a-ad52-1696523129f9-tmp-dir\") pod \"router-with-refs-test-kserve-76f47bf4f8-qtvn6\" (UID: \"a3a55ec7-6060-4d7a-ad52-1696523129f9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"
Sep 17 18:24:19.543173 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.543086    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58dqj\" (UniqueName: \"kubernetes.io/projected/a3a55ec7-6060-4d7a-ad52-1696523129f9-kube-api-access-58dqj\") pod \"router-with-refs-test-kserve-76f47bf4f8-qtvn6\" (UID: \"a3a55ec7-6060-4d7a-ad52-1696523129f9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"
Sep 17 18:24:19.543173 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.543111    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3a55ec7-6060-4d7a-ad52-1696523129f9-dshm\") pod \"router-with-refs-test-kserve-76f47bf4f8-qtvn6\" (UID: \"a3a55ec7-6060-4d7a-ad52-1696523129f9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"
Sep 17 18:24:19.543173 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.543138    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3a55ec7-6060-4d7a-ad52-1696523129f9-tls-certs\") pod \"router-with-refs-test-kserve-76f47bf4f8-qtvn6\" (UID: \"a3a55ec7-6060-4d7a-ad52-1696523129f9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"
Sep 17 18:24:19.543173 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.543160    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3a55ec7-6060-4d7a-ad52-1696523129f9-kserve-provision-location\") pod \"router-with-refs-test-kserve-76f47bf4f8-qtvn6\" (UID: \"a3a55ec7-6060-4d7a-ad52-1696523129f9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"
Sep 17 18:24:19.543522 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.543184    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3a55ec7-6060-4d7a-ad52-1696523129f9-cabundle-cert\") pod \"router-with-refs-test-kserve-76f47bf4f8-qtvn6\" (UID: \"a3a55ec7-6060-4d7a-ad52-1696523129f9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"
Sep 17 18:24:19.543522 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.543407    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3a55ec7-6060-4d7a-ad52-1696523129f9-home\") pod \"router-with-refs-test-kserve-76f47bf4f8-qtvn6\" (UID: \"a3a55ec7-6060-4d7a-ad52-1696523129f9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"
Sep 17 18:24:19.543522 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.543448    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3a55ec7-6060-4d7a-ad52-1696523129f9-model-cache\") pod \"router-with-refs-test-kserve-76f47bf4f8-qtvn6\" (UID: \"a3a55ec7-6060-4d7a-ad52-1696523129f9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"
Sep 17 18:24:19.543723 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.543559    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3a55ec7-6060-4d7a-ad52-1696523129f9-kserve-provision-location\") pod \"router-with-refs-test-kserve-76f47bf4f8-qtvn6\" (UID: \"a3a55ec7-6060-4d7a-ad52-1696523129f9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"
Sep 17 18:24:19.543723 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.543646    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3a55ec7-6060-4d7a-ad52-1696523129f9-tmp-dir\") pod \"router-with-refs-test-kserve-76f47bf4f8-qtvn6\" (UID: \"a3a55ec7-6060-4d7a-ad52-1696523129f9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"
Sep 17 18:24:19.543922 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.543904    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3a55ec7-6060-4d7a-ad52-1696523129f9-cabundle-cert\") pod \"router-with-refs-test-kserve-76f47bf4f8-qtvn6\" (UID: \"a3a55ec7-6060-4d7a-ad52-1696523129f9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"
Sep 17 18:24:19.545375 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.545358    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3a55ec7-6060-4d7a-ad52-1696523129f9-dshm\") pod \"router-with-refs-test-kserve-76f47bf4f8-qtvn6\" (UID: \"a3a55ec7-6060-4d7a-ad52-1696523129f9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"
Sep 17 18:24:19.545785 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.545767    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3a55ec7-6060-4d7a-ad52-1696523129f9-tls-certs\") pod \"router-with-refs-test-kserve-76f47bf4f8-qtvn6\" (UID: \"a3a55ec7-6060-4d7a-ad52-1696523129f9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"
Sep 17 18:24:19.551232 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.551206    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58dqj\" (UniqueName: \"kubernetes.io/projected/a3a55ec7-6060-4d7a-ad52-1696523129f9-kube-api-access-58dqj\") pod \"router-with-refs-test-kserve-76f47bf4f8-qtvn6\" (UID: \"a3a55ec7-6060-4d7a-ad52-1696523129f9\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"
Sep 17 18:24:19.638608 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.638577    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"
Sep 17 18:24:19.916692 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:19.916596    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"]
Sep 17 18:24:19.920251 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:24:19.920229    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3a55ec7_6060_4d7a_ad52_1696523129f9.slice/crio-7ad5427f05353303fd214b491650461cc3c7fa268799a31968d45579fb9d86b9 WatchSource:0}: Error finding container 7ad5427f05353303fd214b491650461cc3c7fa268799a31968d45579fb9d86b9: Status 404 returned error can't find the container with id 7ad5427f05353303fd214b491650461cc3c7fa268799a31968d45579fb9d86b9
Sep 17 18:24:20.095600 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:20.095561    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6" event={"ID":"a3a55ec7-6060-4d7a-ad52-1696523129f9","Type":"ContainerStarted","Data":"5246c4675fdf1818f0ae7a34ec9179e50427dbc5b011cbf0696087db54958728"}
Sep 17 18:24:20.095779 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:20.095618    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6" event={"ID":"a3a55ec7-6060-4d7a-ad52-1696523129f9","Type":"ContainerStarted","Data":"7ad5427f05353303fd214b491650461cc3c7fa268799a31968d45579fb9d86b9"}
Sep 17 18:24:22.105827 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:22.105799    2583 generic.go:356] "Generic (PLEG): container finished" podID="a3a55ec7-6060-4d7a-ad52-1696523129f9" containerID="5246c4675fdf1818f0ae7a34ec9179e50427dbc5b011cbf0696087db54958728" exitCode=0
Sep 17 18:24:22.106160 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:22.105861    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6" event={"ID":"a3a55ec7-6060-4d7a-ad52-1696523129f9","Type":"ContainerDied","Data":"5246c4675fdf1818f0ae7a34ec9179e50427dbc5b011cbf0696087db54958728"}
Sep 17 18:24:23.112986 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:23.112954    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6" event={"ID":"a3a55ec7-6060-4d7a-ad52-1696523129f9","Type":"ContainerStarted","Data":"571e0cd5a04b3e4cdac49c756f53cfe32241696b5ee24182e0e9c4688bb528ef"}
Sep 17 18:24:23.136519 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:23.136458    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6" podStartSLOduration=2.055509263 podStartE2EDuration="4.136443135s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.080933872s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:24:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:24:23.135538471 +0000 UTC m=+2974.226845265" watchObservedRunningTime="2026-09-17 18:24:23.136443135 +0000 UTC m=+2974.227749900"
Sep 17 18:24:23.639495 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:23.639462    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"
Sep 17 18:24:29.639824 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:29.639790    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"
Sep 17 18:24:29.641087 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:29.641063    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6" podUID="a3a55ec7-6060-4d7a-ad52-1696523129f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 17 18:24:39.639973 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:39.639930    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6" podUID="a3a55ec7-6060-4d7a-ad52-1696523129f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 17 18:24:49.639146 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:49.639096    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6" podUID="a3a55ec7-6060-4d7a-ad52-1696523129f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 17 18:24:59.639832 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:24:59.639785    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6" podUID="a3a55ec7-6060-4d7a-ad52-1696523129f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 17 18:25:09.639377 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:25:09.639326    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6" podUID="a3a55ec7-6060-4d7a-ad52-1696523129f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 17 18:25:19.639888 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:25:19.639838    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6" podUID="a3a55ec7-6060-4d7a-ad52-1696523129f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 17 18:25:29.639930 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:25:29.639883    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6" podUID="a3a55ec7-6060-4d7a-ad52-1696523129f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 17 18:25:39.639879 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:25:39.639832    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6" podUID="a3a55ec7-6060-4d7a-ad52-1696523129f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 17 18:25:49.639545 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:25:49.639506    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6" podUID="a3a55ec7-6060-4d7a-ad52-1696523129f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 17 18:25:59.639964 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:25:59.639914    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6" podUID="a3a55ec7-6060-4d7a-ad52-1696523129f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 17 18:26:09.639274 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:09.639227    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6" podUID="a3a55ec7-6060-4d7a-ad52-1696523129f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 17 18:26:19.649230 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:19.649199    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"
Sep 17 18:26:19.657115 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:19.657089    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"
Sep 17 18:26:24.567080 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:24.567044    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"]
Sep 17 18:26:39.589182 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:39.589117    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6" podUID="a3a55ec7-6060-4d7a-ad52-1696523129f9" containerName="main" containerID="cri-o://571e0cd5a04b3e4cdac49c756f53cfe32241696b5ee24182e0e9c4688bb528ef" gracePeriod=45
Sep 17 18:26:39.649869 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:39.649822    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6" podUID="a3a55ec7-6060-4d7a-ad52-1696523129f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 17 18:26:40.339222 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.339195    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"
Sep 17 18:26:40.483145 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.483050    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3a55ec7-6060-4d7a-ad52-1696523129f9-tmp-dir\") pod \"a3a55ec7-6060-4d7a-ad52-1696523129f9\" (UID: \"a3a55ec7-6060-4d7a-ad52-1696523129f9\") "
Sep 17 18:26:40.483145 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.483116    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3a55ec7-6060-4d7a-ad52-1696523129f9-home\") pod \"a3a55ec7-6060-4d7a-ad52-1696523129f9\" (UID: \"a3a55ec7-6060-4d7a-ad52-1696523129f9\") "
Sep 17 18:26:40.483145 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.483149    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3a55ec7-6060-4d7a-ad52-1696523129f9-model-cache\") pod \"a3a55ec7-6060-4d7a-ad52-1696523129f9\" (UID: \"a3a55ec7-6060-4d7a-ad52-1696523129f9\") "
Sep 17 18:26:40.483459 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.483174    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3a55ec7-6060-4d7a-ad52-1696523129f9-tls-certs\") pod \"a3a55ec7-6060-4d7a-ad52-1696523129f9\" (UID: \"a3a55ec7-6060-4d7a-ad52-1696523129f9\") "
Sep 17 18:26:40.483459 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.483192    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3a55ec7-6060-4d7a-ad52-1696523129f9-dshm\") pod \"a3a55ec7-6060-4d7a-ad52-1696523129f9\" (UID: \"a3a55ec7-6060-4d7a-ad52-1696523129f9\") "
Sep 17 18:26:40.483459 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.483212    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3a55ec7-6060-4d7a-ad52-1696523129f9-cabundle-cert\") pod \"a3a55ec7-6060-4d7a-ad52-1696523129f9\" (UID: \"a3a55ec7-6060-4d7a-ad52-1696523129f9\") "
Sep 17 18:26:40.483459 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.483228    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3a55ec7-6060-4d7a-ad52-1696523129f9-kserve-provision-location\") pod \"a3a55ec7-6060-4d7a-ad52-1696523129f9\" (UID: \"a3a55ec7-6060-4d7a-ad52-1696523129f9\") "
Sep 17 18:26:40.483459 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.483256    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58dqj\" (UniqueName: \"kubernetes.io/projected/a3a55ec7-6060-4d7a-ad52-1696523129f9-kube-api-access-58dqj\") pod \"a3a55ec7-6060-4d7a-ad52-1696523129f9\" (UID: \"a3a55ec7-6060-4d7a-ad52-1696523129f9\") "
Sep 17 18:26:40.483759 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.483522    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a55ec7-6060-4d7a-ad52-1696523129f9-model-cache" (OuterVolumeSpecName: "model-cache") pod "a3a55ec7-6060-4d7a-ad52-1696523129f9" (UID: "a3a55ec7-6060-4d7a-ad52-1696523129f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:40.484137 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.484086    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3a55ec7-6060-4d7a-ad52-1696523129f9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a3a55ec7-6060-4d7a-ad52-1696523129f9" (UID: "a3a55ec7-6060-4d7a-ad52-1696523129f9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:26:40.484268 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.484247    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a55ec7-6060-4d7a-ad52-1696523129f9-home" (OuterVolumeSpecName: "home") pod "a3a55ec7-6060-4d7a-ad52-1696523129f9" (UID: "a3a55ec7-6060-4d7a-ad52-1696523129f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:40.485846 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.485823    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3a55ec7-6060-4d7a-ad52-1696523129f9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a3a55ec7-6060-4d7a-ad52-1696523129f9" (UID: "a3a55ec7-6060-4d7a-ad52-1696523129f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:26:40.486017 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.486001    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a55ec7-6060-4d7a-ad52-1696523129f9-dshm" (OuterVolumeSpecName: "dshm") pod "a3a55ec7-6060-4d7a-ad52-1696523129f9" (UID: "a3a55ec7-6060-4d7a-ad52-1696523129f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:40.486080 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.486027    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a55ec7-6060-4d7a-ad52-1696523129f9-kube-api-access-58dqj" (OuterVolumeSpecName: "kube-api-access-58dqj") pod "a3a55ec7-6060-4d7a-ad52-1696523129f9" (UID: "a3a55ec7-6060-4d7a-ad52-1696523129f9"). InnerVolumeSpecName "kube-api-access-58dqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:26:40.501825 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.501779    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a55ec7-6060-4d7a-ad52-1696523129f9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a3a55ec7-6060-4d7a-ad52-1696523129f9" (UID: "a3a55ec7-6060-4d7a-ad52-1696523129f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:40.507912 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.507886    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a55ec7-6060-4d7a-ad52-1696523129f9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a3a55ec7-6060-4d7a-ad52-1696523129f9" (UID: "a3a55ec7-6060-4d7a-ad52-1696523129f9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:40.584115 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.584079    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3a55ec7-6060-4d7a-ad52-1696523129f9-cabundle-cert\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:26:40.584115 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.584109    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3a55ec7-6060-4d7a-ad52-1696523129f9-kserve-provision-location\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:26:40.584115 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.584121    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58dqj\" (UniqueName: \"kubernetes.io/projected/a3a55ec7-6060-4d7a-ad52-1696523129f9-kube-api-access-58dqj\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:26:40.584352 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.584151    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3a55ec7-6060-4d7a-ad52-1696523129f9-tmp-dir\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:26:40.584352 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.584159    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3a55ec7-6060-4d7a-ad52-1696523129f9-home\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:26:40.584352 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.584167    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3a55ec7-6060-4d7a-ad52-1696523129f9-model-cache\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:26:40.584352 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.584176    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3a55ec7-6060-4d7a-ad52-1696523129f9-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:26:40.584352 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.584183    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3a55ec7-6060-4d7a-ad52-1696523129f9-dshm\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:26:40.707893 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.707852    2583 generic.go:356] "Generic (PLEG): container finished" podID="a3a55ec7-6060-4d7a-ad52-1696523129f9" containerID="571e0cd5a04b3e4cdac49c756f53cfe32241696b5ee24182e0e9c4688bb528ef" exitCode=0
Sep 17 18:26:40.708377 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.707898    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"
Sep 17 18:26:40.708377 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.707899    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6" event={"ID":"a3a55ec7-6060-4d7a-ad52-1696523129f9","Type":"ContainerDied","Data":"571e0cd5a04b3e4cdac49c756f53cfe32241696b5ee24182e0e9c4688bb528ef"}
Sep 17 18:26:40.708377 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.707977    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6" event={"ID":"a3a55ec7-6060-4d7a-ad52-1696523129f9","Type":"ContainerDied","Data":"7ad5427f05353303fd214b491650461cc3c7fa268799a31968d45579fb9d86b9"}
Sep 17 18:26:40.708377 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.707999    2583 scope.go:117] "RemoveContainer" containerID="571e0cd5a04b3e4cdac49c756f53cfe32241696b5ee24182e0e9c4688bb528ef"
Sep 17 18:26:40.717731 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.717712    2583 scope.go:117] "RemoveContainer" containerID="5246c4675fdf1818f0ae7a34ec9179e50427dbc5b011cbf0696087db54958728"
Sep 17 18:26:40.725599 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.725578    2583 scope.go:117] "RemoveContainer" containerID="571e0cd5a04b3e4cdac49c756f53cfe32241696b5ee24182e0e9c4688bb528ef"
Sep 17 18:26:40.725956 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:26:40.725933    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"571e0cd5a04b3e4cdac49c756f53cfe32241696b5ee24182e0e9c4688bb528ef\": container with ID starting with 571e0cd5a04b3e4cdac49c756f53cfe32241696b5ee24182e0e9c4688bb528ef not found: ID does not exist" containerID="571e0cd5a04b3e4cdac49c756f53cfe32241696b5ee24182e0e9c4688bb528ef"
Sep 17 18:26:40.726066 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.725964    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"571e0cd5a04b3e4cdac49c756f53cfe32241696b5ee24182e0e9c4688bb528ef"} err="failed to get container status \"571e0cd5a04b3e4cdac49c756f53cfe32241696b5ee24182e0e9c4688bb528ef\": rpc error: code = NotFound desc = could not find container \"571e0cd5a04b3e4cdac49c756f53cfe32241696b5ee24182e0e9c4688bb528ef\": container with ID starting with 571e0cd5a04b3e4cdac49c756f53cfe32241696b5ee24182e0e9c4688bb528ef not found: ID does not exist"
Sep 17 18:26:40.726066 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.725982    2583 scope.go:117] "RemoveContainer" containerID="5246c4675fdf1818f0ae7a34ec9179e50427dbc5b011cbf0696087db54958728"
Sep 17 18:26:40.726307 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:26:40.726286    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5246c4675fdf1818f0ae7a34ec9179e50427dbc5b011cbf0696087db54958728\": container with ID starting with 5246c4675fdf1818f0ae7a34ec9179e50427dbc5b011cbf0696087db54958728 not found: ID does not exist" containerID="5246c4675fdf1818f0ae7a34ec9179e50427dbc5b011cbf0696087db54958728"
Sep 17 18:26:40.726357 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.726314    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5246c4675fdf1818f0ae7a34ec9179e50427dbc5b011cbf0696087db54958728"} err="failed to get container status \"5246c4675fdf1818f0ae7a34ec9179e50427dbc5b011cbf0696087db54958728\": rpc error: code = NotFound desc = could not find container \"5246c4675fdf1818f0ae7a34ec9179e50427dbc5b011cbf0696087db54958728\": container with ID starting with 5246c4675fdf1818f0ae7a34ec9179e50427dbc5b011cbf0696087db54958728 not found: ID does not exist"
Sep 17 18:26:40.730875 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.730851    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"]
Sep 17 18:26:40.736982 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:40.736914    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-76f47bf4f8-qtvn6"]
Sep 17 18:26:41.039570 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:26:41.039489    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3a55ec7-6060-4d7a-ad52-1696523129f9" path="/var/lib/kubelet/pods/a3a55ec7-6060-4d7a-ad52-1696523129f9/volumes"
Sep 17 18:27:09.227895 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.227816    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"]
Sep 17 18:27:09.228377 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.228091    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3a55ec7-6060-4d7a-ad52-1696523129f9" containerName="storage-initializer"
Sep 17 18:27:09.228377 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.228102    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3a55ec7-6060-4d7a-ad52-1696523129f9" containerName="storage-initializer"
Sep 17 18:27:09.228377 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.228113    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3a55ec7-6060-4d7a-ad52-1696523129f9" containerName="main"
Sep 17 18:27:09.228377 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.228119    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3a55ec7-6060-4d7a-ad52-1696523129f9" containerName="main"
Sep 17 18:27:09.228377 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.228170    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3a55ec7-6060-4d7a-ad52-1696523129f9" containerName="main"
Sep 17 18:27:09.231208 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.231184    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"
Sep 17 18:27:09.233456 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.233436    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:27:09.234038 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.234014    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 17 18:27:09.234238 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.234222    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 17 18:27:09.234319 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.234298    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-jhjnz\""
Sep 17 18:27:09.234390 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.234338    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 17 18:27:09.234463 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.234387    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 17 18:27:09.244478 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.244453    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"]
Sep 17 18:27:09.309578 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.309550    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/202e8856-86a5-4a0a-9462-7b7dd3f2b767-tls-certs\") pod \"stop-feature-test-kserve-c54f4856d-llpkl\" (UID: \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"
Sep 17 18:27:09.309732 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.309584    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/202e8856-86a5-4a0a-9462-7b7dd3f2b767-cabundle-cert\") pod \"stop-feature-test-kserve-c54f4856d-llpkl\" (UID: \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"
Sep 17 18:27:09.309732 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.309608    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/202e8856-86a5-4a0a-9462-7b7dd3f2b767-tmp-dir\") pod \"stop-feature-test-kserve-c54f4856d-llpkl\" (UID: \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"
Sep 17 18:27:09.309732 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.309648    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/202e8856-86a5-4a0a-9462-7b7dd3f2b767-home\") pod \"stop-feature-test-kserve-c54f4856d-llpkl\" (UID: \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"
Sep 17 18:27:09.309732 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.309723    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/202e8856-86a5-4a0a-9462-7b7dd3f2b767-kserve-provision-location\") pod \"stop-feature-test-kserve-c54f4856d-llpkl\" (UID: \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"
Sep 17 18:27:09.309908 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.309750    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/202e8856-86a5-4a0a-9462-7b7dd3f2b767-dshm\") pod \"stop-feature-test-kserve-c54f4856d-llpkl\" (UID: \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"
Sep 17 18:27:09.309908 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.309777    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-876xl\" (UniqueName: \"kubernetes.io/projected/202e8856-86a5-4a0a-9462-7b7dd3f2b767-kube-api-access-876xl\") pod \"stop-feature-test-kserve-c54f4856d-llpkl\" (UID: \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"
Sep 17 18:27:09.309908 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.309797    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/202e8856-86a5-4a0a-9462-7b7dd3f2b767-model-cache\") pod \"stop-feature-test-kserve-c54f4856d-llpkl\" (UID: \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"
Sep 17 18:27:09.398669 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.398628    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn"]
Sep 17 18:27:09.401901 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.401882    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn"
Sep 17 18:27:09.404179 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.404152    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-f9w5p\""
Sep 17 18:27:09.410327 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.410182    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/202e8856-86a5-4a0a-9462-7b7dd3f2b767-tmp-dir\") pod \"stop-feature-test-kserve-c54f4856d-llpkl\" (UID: \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"
Sep 17 18:27:09.410327 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.410219    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/202e8856-86a5-4a0a-9462-7b7dd3f2b767-home\") pod \"stop-feature-test-kserve-c54f4856d-llpkl\" (UID: \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"
Sep 17 18:27:09.410327 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.410256    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64ac746a-f9f0-4e75-94a9-94ba5d0941af-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn\" (UID: \"64ac746a-f9f0-4e75-94a9-94ba5d0941af\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn"
Sep 17 18:27:09.410327 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.410294    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/202e8856-86a5-4a0a-9462-7b7dd3f2b767-kserve-provision-location\") pod \"stop-feature-test-kserve-c54f4856d-llpkl\" (UID: \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"
Sep 17 18:27:09.410327 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.410318    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s68jw\" (UniqueName: \"kubernetes.io/projected/64ac746a-f9f0-4e75-94a9-94ba5d0941af-kube-api-access-s68jw\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn\" (UID: \"64ac746a-f9f0-4e75-94a9-94ba5d0941af\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn"
Sep 17 18:27:09.410609 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.410361    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/202e8856-86a5-4a0a-9462-7b7dd3f2b767-dshm\") pod \"stop-feature-test-kserve-c54f4856d-llpkl\" (UID: \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"
Sep 17 18:27:09.410609 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.410393    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-876xl\" (UniqueName: \"kubernetes.io/projected/202e8856-86a5-4a0a-9462-7b7dd3f2b767-kube-api-access-876xl\") pod \"stop-feature-test-kserve-c54f4856d-llpkl\" (UID: \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"
Sep 17 18:27:09.410609 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.410418    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/202e8856-86a5-4a0a-9462-7b7dd3f2b767-model-cache\") pod \"stop-feature-test-kserve-c54f4856d-llpkl\" (UID: \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"
Sep 17 18:27:09.410609 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.410461    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/202e8856-86a5-4a0a-9462-7b7dd3f2b767-tls-certs\") pod \"stop-feature-test-kserve-c54f4856d-llpkl\" (UID: \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"
Sep 17 18:27:09.410609 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.410491    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/202e8856-86a5-4a0a-9462-7b7dd3f2b767-cabundle-cert\") pod \"stop-feature-test-kserve-c54f4856d-llpkl\" (UID: \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"
Sep 17 18:27:09.410897 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.410628    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/202e8856-86a5-4a0a-9462-7b7dd3f2b767-tmp-dir\") pod \"stop-feature-test-kserve-c54f4856d-llpkl\" (UID: \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"
Sep 17 18:27:09.410958 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.410905    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/202e8856-86a5-4a0a-9462-7b7dd3f2b767-home\") pod \"stop-feature-test-kserve-c54f4856d-llpkl\" (UID: \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"
Sep 17 18:27:09.411193 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.411141    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/202e8856-86a5-4a0a-9462-7b7dd3f2b767-kserve-provision-location\") pod \"stop-feature-test-kserve-c54f4856d-llpkl\" (UID: \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"
Sep 17 18:27:09.411299 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.411201    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/202e8856-86a5-4a0a-9462-7b7dd3f2b767-cabundle-cert\") pod \"stop-feature-test-kserve-c54f4856d-llpkl\" (UID: \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"
Sep 17 18:27:09.411484 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.411451    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn"]
Sep 17 18:27:09.411567 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.411458    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/202e8856-86a5-4a0a-9462-7b7dd3f2b767-model-cache\") pod \"stop-feature-test-kserve-c54f4856d-llpkl\" (UID: \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"
Sep 17 18:27:09.413965 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.413942    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/202e8856-86a5-4a0a-9462-7b7dd3f2b767-dshm\") pod \"stop-feature-test-kserve-c54f4856d-llpkl\" (UID: \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"
Sep 17 18:27:09.414275 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.414253    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/202e8856-86a5-4a0a-9462-7b7dd3f2b767-tls-certs\") pod \"stop-feature-test-kserve-c54f4856d-llpkl\" (UID: \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"
Sep 17 18:27:09.419460 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.419432    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-876xl\" (UniqueName: \"kubernetes.io/projected/202e8856-86a5-4a0a-9462-7b7dd3f2b767-kube-api-access-876xl\") pod \"stop-feature-test-kserve-c54f4856d-llpkl\" (UID: \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"
Sep 17 18:27:09.511529 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.511442    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64ac746a-f9f0-4e75-94a9-94ba5d0941af-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn\" (UID: \"64ac746a-f9f0-4e75-94a9-94ba5d0941af\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn"
Sep 17 18:27:09.511529 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.511487    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s68jw\" (UniqueName: \"kubernetes.io/projected/64ac746a-f9f0-4e75-94a9-94ba5d0941af-kube-api-access-s68jw\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn\" (UID: \"64ac746a-f9f0-4e75-94a9-94ba5d0941af\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn"
Sep 17 18:27:09.513948 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.513929    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64ac746a-f9f0-4e75-94a9-94ba5d0941af-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn\" (UID: \"64ac746a-f9f0-4e75-94a9-94ba5d0941af\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn"
Sep 17 18:27:09.519826 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.519799    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s68jw\" (UniqueName: \"kubernetes.io/projected/64ac746a-f9f0-4e75-94a9-94ba5d0941af-kube-api-access-s68jw\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn\" (UID: \"64ac746a-f9f0-4e75-94a9-94ba5d0941af\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn"
Sep 17 18:27:09.541414 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.541392    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"
Sep 17 18:27:09.715717 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.715692    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn"
Sep 17 18:27:09.838704 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.838538    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"]
Sep 17 18:27:09.841231 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:27:09.841204    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod202e8856_86a5_4a0a_9462_7b7dd3f2b767.slice/crio-42be3e7994fc7731ab6487152ce08eff98ead9b1baa343e428ecc5d71dbb81ae WatchSource:0}: Error finding container 42be3e7994fc7731ab6487152ce08eff98ead9b1baa343e428ecc5d71dbb81ae: Status 404 returned error can't find the container with id 42be3e7994fc7731ab6487152ce08eff98ead9b1baa343e428ecc5d71dbb81ae
Sep 17 18:27:09.985320 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:09.985258    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn"]
Sep 17 18:27:09.988192 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:27:09.988163    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ac746a_f9f0_4e75_94a9_94ba5d0941af.slice/crio-bcc76e78cc14050662345f129d928f6cd5ce045e6837a25182edcc80b6474b48 WatchSource:0}: Error finding container bcc76e78cc14050662345f129d928f6cd5ce045e6837a25182edcc80b6474b48: Status 404 returned error can't find the container with id bcc76e78cc14050662345f129d928f6cd5ce045e6837a25182edcc80b6474b48
Sep 17 18:27:10.835887 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:10.835852    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn" event={"ID":"64ac746a-f9f0-4e75-94a9-94ba5d0941af","Type":"ContainerStarted","Data":"a29213e52eefee0a69abd1b0423fa169626b16ebd37c851da131eb8af8555d72"}
Sep 17 18:27:10.836358 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:10.835906    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn"
Sep 17 18:27:10.836358 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:10.835928    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn" event={"ID":"64ac746a-f9f0-4e75-94a9-94ba5d0941af","Type":"ContainerStarted","Data":"bcc76e78cc14050662345f129d928f6cd5ce045e6837a25182edcc80b6474b48"}
Sep 17 18:27:10.837383 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:10.837359    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl" event={"ID":"202e8856-86a5-4a0a-9462-7b7dd3f2b767","Type":"ContainerStarted","Data":"def9b67b6b9129cb1c38d31f1bf95ab9436a70532a6f1f1e2d7ea3b8a27225d8"}
Sep 17 18:27:10.837499 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:10.837406    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl" event={"ID":"202e8856-86a5-4a0a-9462-7b7dd3f2b767","Type":"ContainerStarted","Data":"42be3e7994fc7731ab6487152ce08eff98ead9b1baa343e428ecc5d71dbb81ae"}
Sep 17 18:27:10.853940 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:10.853898    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn" podStartSLOduration=1.853884901 podStartE2EDuration="1.853884901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:27:10.852578252 +0000 UTC m=+3141.943885046" watchObservedRunningTime="2026-09-17 18:27:10.853884901 +0000 UTC m=+3141.945191665"
Sep 17 18:27:12.846598 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:12.846566    2583 generic.go:356] "Generic (PLEG): container finished" podID="202e8856-86a5-4a0a-9462-7b7dd3f2b767" containerID="def9b67b6b9129cb1c38d31f1bf95ab9436a70532a6f1f1e2d7ea3b8a27225d8" exitCode=0
Sep 17 18:27:12.847005 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:12.846602    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl" event={"ID":"202e8856-86a5-4a0a-9462-7b7dd3f2b767","Type":"ContainerDied","Data":"def9b67b6b9129cb1c38d31f1bf95ab9436a70532a6f1f1e2d7ea3b8a27225d8"}
Sep 17 18:27:13.853152 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:13.853109    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl" event={"ID":"202e8856-86a5-4a0a-9462-7b7dd3f2b767","Type":"ContainerStarted","Data":"f137bfc37458d9cf57345a12a3f6be5cef61a73131d474b53ee9a3899c36b337"}
Sep 17 18:27:13.877627 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:13.877559    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl" podStartSLOduration=2.7287057839999997 podStartE2EDuration="4.877536723s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.148830939s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:27:13.875543855 +0000 UTC m=+3144.966850652" watchObservedRunningTime="2026-09-17 18:27:13.877536723 +0000 UTC m=+3144.968843489"
Sep 17 18:27:14.542537 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:14.542498    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"
Sep 17 18:27:19.541825 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:19.541792    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"
Sep 17 18:27:19.543174 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:19.543148    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl" podUID="202e8856-86a5-4a0a-9462-7b7dd3f2b767" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 18:27:29.542390 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:29.542347    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl" podUID="202e8856-86a5-4a0a-9462-7b7dd3f2b767" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 18:27:34.859508 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:34.859477    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"]
Sep 17 18:27:34.862834 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:34.862815    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"
Sep 17 18:27:34.865691 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:34.865649    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 17 18:27:34.865941 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:34.865648    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 17 18:27:34.866058 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:34.865941    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 17 18:27:34.866058 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:34.865730    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-v6v88\""
Sep 17 18:27:34.866058 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:34.865734    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:27:34.866058 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:34.865782    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 17 18:27:34.875811 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:34.875789    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"]
Sep 17 18:27:34.919769 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:34.919740    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/631487f2-a57f-4dd0-91c2-4d6bab476d53-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x\" (UID: \"631487f2-a57f-4dd0-91c2-4d6bab476d53\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"
Sep 17 18:27:34.919905 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:34.919779    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/631487f2-a57f-4dd0-91c2-4d6bab476d53-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x\" (UID: \"631487f2-a57f-4dd0-91c2-4d6bab476d53\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"
Sep 17 18:27:34.919905 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:34.919809    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/631487f2-a57f-4dd0-91c2-4d6bab476d53-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x\" (UID: \"631487f2-a57f-4dd0-91c2-4d6bab476d53\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"
Sep 17 18:27:34.919905 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:34.919838    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/631487f2-a57f-4dd0-91c2-4d6bab476d53-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x\" (UID: \"631487f2-a57f-4dd0-91c2-4d6bab476d53\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"
Sep 17 18:27:34.919905 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:34.919866    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/631487f2-a57f-4dd0-91c2-4d6bab476d53-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x\" (UID: \"631487f2-a57f-4dd0-91c2-4d6bab476d53\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"
Sep 17 18:27:34.920075 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:34.919905    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxs6x\" (UniqueName: \"kubernetes.io/projected/631487f2-a57f-4dd0-91c2-4d6bab476d53-kube-api-access-bxs6x\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x\" (UID: \"631487f2-a57f-4dd0-91c2-4d6bab476d53\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"
Sep 17 18:27:34.920075 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:34.919926    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/631487f2-a57f-4dd0-91c2-4d6bab476d53-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x\" (UID: \"631487f2-a57f-4dd0-91c2-4d6bab476d53\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"
Sep 17 18:27:34.920075 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:34.919960    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/631487f2-a57f-4dd0-91c2-4d6bab476d53-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x\" (UID: \"631487f2-a57f-4dd0-91c2-4d6bab476d53\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"
Sep 17 18:27:35.021022 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:35.020985    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxs6x\" (UniqueName: \"kubernetes.io/projected/631487f2-a57f-4dd0-91c2-4d6bab476d53-kube-api-access-bxs6x\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x\" (UID: \"631487f2-a57f-4dd0-91c2-4d6bab476d53\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"
Sep 17 18:27:35.021022 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:35.021030    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/631487f2-a57f-4dd0-91c2-4d6bab476d53-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x\" (UID: \"631487f2-a57f-4dd0-91c2-4d6bab476d53\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"
Sep 17 18:27:35.021301 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:35.021077    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/631487f2-a57f-4dd0-91c2-4d6bab476d53-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x\" (UID: \"631487f2-a57f-4dd0-91c2-4d6bab476d53\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"
Sep 17 18:27:35.021301 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:35.021121    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/631487f2-a57f-4dd0-91c2-4d6bab476d53-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x\" (UID: \"631487f2-a57f-4dd0-91c2-4d6bab476d53\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"
Sep 17 18:27:35.021301 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:35.021154    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/631487f2-a57f-4dd0-91c2-4d6bab476d53-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x\" (UID: \"631487f2-a57f-4dd0-91c2-4d6bab476d53\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"
Sep 17 18:27:35.021301 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:35.021189    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/631487f2-a57f-4dd0-91c2-4d6bab476d53-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x\" (UID: \"631487f2-a57f-4dd0-91c2-4d6bab476d53\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"
Sep 17 18:27:35.021301 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:35.021209    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/631487f2-a57f-4dd0-91c2-4d6bab476d53-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x\" (UID: \"631487f2-a57f-4dd0-91c2-4d6bab476d53\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"
Sep 17 18:27:35.021301 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:35.021242    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/631487f2-a57f-4dd0-91c2-4d6bab476d53-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x\" (UID: \"631487f2-a57f-4dd0-91c2-4d6bab476d53\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"
Sep 17 18:27:35.021588 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:35.021484    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/631487f2-a57f-4dd0-91c2-4d6bab476d53-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x\" (UID: \"631487f2-a57f-4dd0-91c2-4d6bab476d53\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"
Sep 17 18:27:35.021697 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:35.021645    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/631487f2-a57f-4dd0-91c2-4d6bab476d53-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x\" (UID: \"631487f2-a57f-4dd0-91c2-4d6bab476d53\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"
Sep 17 18:27:35.021830 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:35.021810    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/631487f2-a57f-4dd0-91c2-4d6bab476d53-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x\" (UID: \"631487f2-a57f-4dd0-91c2-4d6bab476d53\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"
Sep 17 18:27:35.021830 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:35.021823    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/631487f2-a57f-4dd0-91c2-4d6bab476d53-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x\" (UID: \"631487f2-a57f-4dd0-91c2-4d6bab476d53\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"
Sep 17 18:27:35.021987 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:35.021969    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/631487f2-a57f-4dd0-91c2-4d6bab476d53-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x\" (UID: \"631487f2-a57f-4dd0-91c2-4d6bab476d53\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"
Sep 17 18:27:35.023677 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:35.023647    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/631487f2-a57f-4dd0-91c2-4d6bab476d53-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x\" (UID: \"631487f2-a57f-4dd0-91c2-4d6bab476d53\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"
Sep 17 18:27:35.024097 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:35.024078    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/631487f2-a57f-4dd0-91c2-4d6bab476d53-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x\" (UID: \"631487f2-a57f-4dd0-91c2-4d6bab476d53\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"
Sep 17 18:27:35.029514 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:35.029487    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxs6x\" (UniqueName: \"kubernetes.io/projected/631487f2-a57f-4dd0-91c2-4d6bab476d53-kube-api-access-bxs6x\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x\" (UID: \"631487f2-a57f-4dd0-91c2-4d6bab476d53\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"
Sep 17 18:27:35.173265 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:35.173160    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"
Sep 17 18:27:35.463833 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:35.463805    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"]
Sep 17 18:27:35.465095 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:27:35.465067    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod631487f2_a57f_4dd0_91c2_4d6bab476d53.slice/crio-dbdd3c42a610600d592cfff9d1de00c1ef854c8ce57a90868c5b1f4c6662b65c WatchSource:0}: Error finding container dbdd3c42a610600d592cfff9d1de00c1ef854c8ce57a90868c5b1f4c6662b65c: Status 404 returned error can't find the container with id dbdd3c42a610600d592cfff9d1de00c1ef854c8ce57a90868c5b1f4c6662b65c
Sep 17 18:27:35.957241 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:35.957204    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x" event={"ID":"631487f2-a57f-4dd0-91c2-4d6bab476d53","Type":"ContainerStarted","Data":"850a6329e0a180d82fd60f4d7d5d3a30cdfbff34ede63718208fe978451bd0fb"}
Sep 17 18:27:35.957642 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:35.957256    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x" event={"ID":"631487f2-a57f-4dd0-91c2-4d6bab476d53","Type":"ContainerStarted","Data":"dbdd3c42a610600d592cfff9d1de00c1ef854c8ce57a90868c5b1f4c6662b65c"}
Sep 17 18:27:37.967774 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:37.967737    2583 generic.go:356] "Generic (PLEG): container finished" podID="631487f2-a57f-4dd0-91c2-4d6bab476d53" containerID="850a6329e0a180d82fd60f4d7d5d3a30cdfbff34ede63718208fe978451bd0fb" exitCode=0
Sep 17 18:27:37.968262 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:37.967807    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x" event={"ID":"631487f2-a57f-4dd0-91c2-4d6bab476d53","Type":"ContainerDied","Data":"850a6329e0a180d82fd60f4d7d5d3a30cdfbff34ede63718208fe978451bd0fb"}
Sep 17 18:27:38.974247 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:38.974207    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x" event={"ID":"631487f2-a57f-4dd0-91c2-4d6bab476d53","Type":"ContainerStarted","Data":"d7cff33f2af4e6d53d771172ad7aa5be516f1605d93f026816f0fc05f5f11722"}
Sep 17 18:27:39.001188 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:39.001133    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x" podStartSLOduration=2.6864622320000002 podStartE2EDuration="5.001118407s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.314656175s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:27:38.998838601 +0000 UTC m=+3170.090145377" watchObservedRunningTime="2026-09-17 18:27:39.001118407 +0000 UTC m=+3170.092425171"
Sep 17 18:27:39.542908 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:39.542860    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl" podUID="202e8856-86a5-4a0a-9462-7b7dd3f2b767" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 18:27:41.842909 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:41.842876    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn"
Sep 17 18:27:45.173643 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:45.173595    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"
Sep 17 18:27:45.174155 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:45.173701    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"
Sep 17 18:27:45.175107 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:45.175075    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x" podUID="631487f2-a57f-4dd0-91c2-4d6bab476d53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 17 18:27:49.542945 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:49.542899    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl" podUID="202e8856-86a5-4a0a-9462-7b7dd3f2b767" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 18:27:55.174125 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:55.174078    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x" podUID="631487f2-a57f-4dd0-91c2-4d6bab476d53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 17 18:27:59.542246 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:27:59.542193    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl" podUID="202e8856-86a5-4a0a-9462-7b7dd3f2b767" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 18:28:05.174421 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:28:05.174370    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x" podUID="631487f2-a57f-4dd0-91c2-4d6bab476d53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 17 18:28:09.541905 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:28:09.541849    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl" podUID="202e8856-86a5-4a0a-9462-7b7dd3f2b767" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 18:28:15.174119 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:28:15.174075    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x" podUID="631487f2-a57f-4dd0-91c2-4d6bab476d53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 17 18:28:19.542559 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:28:19.542508    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl" podUID="202e8856-86a5-4a0a-9462-7b7dd3f2b767" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 18:28:25.173763 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:28:25.173723    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x" podUID="631487f2-a57f-4dd0-91c2-4d6bab476d53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 17 18:28:29.542177 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:28:29.542137    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl" podUID="202e8856-86a5-4a0a-9462-7b7dd3f2b767" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 18:28:35.174305 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:28:35.174253    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x" podUID="631487f2-a57f-4dd0-91c2-4d6bab476d53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 17 18:28:39.541967 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:28:39.541918    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl" podUID="202e8856-86a5-4a0a-9462-7b7dd3f2b767" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 18:28:45.174595 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:28:45.174556    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x" podUID="631487f2-a57f-4dd0-91c2-4d6bab476d53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 17 18:28:49.542480 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:28:49.542435    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl" podUID="202e8856-86a5-4a0a-9462-7b7dd3f2b767" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 18:28:55.174301 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:28:55.174258    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x" podUID="631487f2-a57f-4dd0-91c2-4d6bab476d53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 17 18:28:59.542850 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:28:59.542805    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl" podUID="202e8856-86a5-4a0a-9462-7b7dd3f2b767" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 18:29:05.174608 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:05.174550    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x" podUID="631487f2-a57f-4dd0-91c2-4d6bab476d53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 17 18:29:09.542534 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:09.542489    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl" podUID="202e8856-86a5-4a0a-9462-7b7dd3f2b767" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 18:29:15.173995 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:15.173948    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x" podUID="631487f2-a57f-4dd0-91c2-4d6bab476d53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 17 18:29:19.542298 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:19.542252    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl" podUID="202e8856-86a5-4a0a-9462-7b7dd3f2b767" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 18:29:25.173931 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:25.173881    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x" podUID="631487f2-a57f-4dd0-91c2-4d6bab476d53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 17 18:29:29.542145 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:29.542100    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl" podUID="202e8856-86a5-4a0a-9462-7b7dd3f2b767" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 18:29:35.174432 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:35.174381    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x" podUID="631487f2-a57f-4dd0-91c2-4d6bab476d53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 17 18:29:39.551729 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:39.551690    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"
Sep 17 18:29:39.559630 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:39.559605    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"
Sep 17 18:29:40.948721 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:40.948679    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn"]
Sep 17 18:29:40.952049 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:40.952019    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"]
Sep 17 18:29:41.842959 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:41.842925    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn" podUID="64ac746a-f9f0-4e75-94a9-94ba5d0941af" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:29:45.184293 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:45.184263    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"
Sep 17 18:29:45.191750 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:45.191727    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"
Sep 17 18:29:51.843489 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:51.843452    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn" podUID="64ac746a-f9f0-4e75-94a9-94ba5d0941af" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:29:53.864318 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:53.864279    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"]
Sep 17 18:29:55.973057 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:55.973016    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn" podUID="64ac746a-f9f0-4e75-94a9-94ba5d0941af" containerName="main" containerID="cri-o://a29213e52eefee0a69abd1b0423fa169626b16ebd37c851da131eb8af8555d72" gracePeriod=45
Sep 17 18:29:55.982064 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:55.982029    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl" podUID="202e8856-86a5-4a0a-9462-7b7dd3f2b767" containerName="main" containerID="cri-o://f137bfc37458d9cf57345a12a3f6be5cef61a73131d474b53ee9a3899c36b337" gracePeriod=45
Sep 17 18:29:56.552478 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:56.552431    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl" podUID="202e8856-86a5-4a0a-9462-7b7dd3f2b767" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 17 18:29:56.636697 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:56.636640    2583 generic.go:356] "Generic (PLEG): container finished" podID="202e8856-86a5-4a0a-9462-7b7dd3f2b767" containerID="f137bfc37458d9cf57345a12a3f6be5cef61a73131d474b53ee9a3899c36b337" exitCode=0
Sep 17 18:29:56.636831 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:56.636696    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl" event={"ID":"202e8856-86a5-4a0a-9462-7b7dd3f2b767","Type":"ContainerDied","Data":"f137bfc37458d9cf57345a12a3f6be5cef61a73131d474b53ee9a3899c36b337"}
Sep 17 18:29:56.653406 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:56.653383    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"
Sep 17 18:29:56.742939 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:56.742859    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/202e8856-86a5-4a0a-9462-7b7dd3f2b767-home\") pod \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\" (UID: \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\") "
Sep 17 18:29:56.742939 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:56.742889    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/202e8856-86a5-4a0a-9462-7b7dd3f2b767-dshm\") pod \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\" (UID: \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\") "
Sep 17 18:29:56.742939 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:56.742919    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/202e8856-86a5-4a0a-9462-7b7dd3f2b767-tmp-dir\") pod \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\" (UID: \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\") "
Sep 17 18:29:56.743221 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:56.742962    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/202e8856-86a5-4a0a-9462-7b7dd3f2b767-kserve-provision-location\") pod \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\" (UID: \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\") "
Sep 17 18:29:56.743221 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:56.742989    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/202e8856-86a5-4a0a-9462-7b7dd3f2b767-tls-certs\") pod \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\" (UID: \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\") "
Sep 17 18:29:56.743221 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:56.743009    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/202e8856-86a5-4a0a-9462-7b7dd3f2b767-cabundle-cert\") pod \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\" (UID: \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\") "
Sep 17 18:29:56.743221 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:56.743029    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/202e8856-86a5-4a0a-9462-7b7dd3f2b767-model-cache\") pod \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\" (UID: \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\") "
Sep 17 18:29:56.743221 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:56.743064    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-876xl\" (UniqueName: \"kubernetes.io/projected/202e8856-86a5-4a0a-9462-7b7dd3f2b767-kube-api-access-876xl\") pod \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\" (UID: \"202e8856-86a5-4a0a-9462-7b7dd3f2b767\") "
Sep 17 18:29:56.743582 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:56.743512    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/202e8856-86a5-4a0a-9462-7b7dd3f2b767-model-cache" (OuterVolumeSpecName: "model-cache") pod "202e8856-86a5-4a0a-9462-7b7dd3f2b767" (UID: "202e8856-86a5-4a0a-9462-7b7dd3f2b767"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:56.743582 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:56.743521    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/202e8856-86a5-4a0a-9462-7b7dd3f2b767-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "202e8856-86a5-4a0a-9462-7b7dd3f2b767" (UID: "202e8856-86a5-4a0a-9462-7b7dd3f2b767"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:29:56.743805 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:56.743681    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/202e8856-86a5-4a0a-9462-7b7dd3f2b767-home" (OuterVolumeSpecName: "home") pod "202e8856-86a5-4a0a-9462-7b7dd3f2b767" (UID: "202e8856-86a5-4a0a-9462-7b7dd3f2b767"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:56.745364 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:56.745331    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/202e8856-86a5-4a0a-9462-7b7dd3f2b767-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "202e8856-86a5-4a0a-9462-7b7dd3f2b767" (UID: "202e8856-86a5-4a0a-9462-7b7dd3f2b767"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:29:56.745777 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:56.745747    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/202e8856-86a5-4a0a-9462-7b7dd3f2b767-kube-api-access-876xl" (OuterVolumeSpecName: "kube-api-access-876xl") pod "202e8856-86a5-4a0a-9462-7b7dd3f2b767" (UID: "202e8856-86a5-4a0a-9462-7b7dd3f2b767"). InnerVolumeSpecName "kube-api-access-876xl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:29:56.745867 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:56.745796    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/202e8856-86a5-4a0a-9462-7b7dd3f2b767-dshm" (OuterVolumeSpecName: "dshm") pod "202e8856-86a5-4a0a-9462-7b7dd3f2b767" (UID: "202e8856-86a5-4a0a-9462-7b7dd3f2b767"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:56.756311 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:56.756281    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/202e8856-86a5-4a0a-9462-7b7dd3f2b767-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "202e8856-86a5-4a0a-9462-7b7dd3f2b767" (UID: "202e8856-86a5-4a0a-9462-7b7dd3f2b767"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:56.767379 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:56.767347    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/202e8856-86a5-4a0a-9462-7b7dd3f2b767-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "202e8856-86a5-4a0a-9462-7b7dd3f2b767" (UID: "202e8856-86a5-4a0a-9462-7b7dd3f2b767"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:56.844054 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:56.844025    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/202e8856-86a5-4a0a-9462-7b7dd3f2b767-home\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:29:56.844054 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:56.844051    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/202e8856-86a5-4a0a-9462-7b7dd3f2b767-dshm\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:29:56.844220 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:56.844080    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/202e8856-86a5-4a0a-9462-7b7dd3f2b767-tmp-dir\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:29:56.844220 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:56.844091    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/202e8856-86a5-4a0a-9462-7b7dd3f2b767-kserve-provision-location\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:29:56.844220 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:56.844101    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/202e8856-86a5-4a0a-9462-7b7dd3f2b767-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:29:56.844220 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:56.844109    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/202e8856-86a5-4a0a-9462-7b7dd3f2b767-cabundle-cert\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:29:56.844220 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:56.844117    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/202e8856-86a5-4a0a-9462-7b7dd3f2b767-model-cache\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:29:56.844220 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:56.844124    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-876xl\" (UniqueName: \"kubernetes.io/projected/202e8856-86a5-4a0a-9462-7b7dd3f2b767-kube-api-access-876xl\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:29:57.642853 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:57.642826    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"
Sep 17 18:29:57.643349 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:57.642826    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl" event={"ID":"202e8856-86a5-4a0a-9462-7b7dd3f2b767","Type":"ContainerDied","Data":"42be3e7994fc7731ab6487152ce08eff98ead9b1baa343e428ecc5d71dbb81ae"}
Sep 17 18:29:57.643349 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:57.642957    2583 scope.go:117] "RemoveContainer" containerID="f137bfc37458d9cf57345a12a3f6be5cef61a73131d474b53ee9a3899c36b337"
Sep 17 18:29:57.651371 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:57.651345    2583 scope.go:117] "RemoveContainer" containerID="def9b67b6b9129cb1c38d31f1bf95ab9436a70532a6f1f1e2d7ea3b8a27225d8"
Sep 17 18:29:57.663551 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:57.663173    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"]
Sep 17 18:29:57.666569 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:57.666544    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-c54f4856d-llpkl"]
Sep 17 18:29:59.040582 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:29:59.040548    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="202e8856-86a5-4a0a-9462-7b7dd3f2b767" path="/var/lib/kubelet/pods/202e8856-86a5-4a0a-9462-7b7dd3f2b767/volumes"
Sep 17 18:30:01.843315 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:01.843267    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn" podUID="64ac746a-f9f0-4e75-94a9-94ba5d0941af" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:30:01.843818 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:01.843390    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn"
Sep 17 18:30:08.888251 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:08.888212    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x" podUID="631487f2-a57f-4dd0-91c2-4d6bab476d53" containerName="main" containerID="cri-o://d7cff33f2af4e6d53d771172ad7aa5be516f1605d93f026816f0fc05f5f11722" gracePeriod=45
Sep 17 18:30:09.544842 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.544819    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"
Sep 17 18:30:09.647467 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.647384    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxs6x\" (UniqueName: \"kubernetes.io/projected/631487f2-a57f-4dd0-91c2-4d6bab476d53-kube-api-access-bxs6x\") pod \"631487f2-a57f-4dd0-91c2-4d6bab476d53\" (UID: \"631487f2-a57f-4dd0-91c2-4d6bab476d53\") "
Sep 17 18:30:09.647467 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.647428    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/631487f2-a57f-4dd0-91c2-4d6bab476d53-model-cache\") pod \"631487f2-a57f-4dd0-91c2-4d6bab476d53\" (UID: \"631487f2-a57f-4dd0-91c2-4d6bab476d53\") "
Sep 17 18:30:09.647720 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.647471    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/631487f2-a57f-4dd0-91c2-4d6bab476d53-kserve-provision-location\") pod \"631487f2-a57f-4dd0-91c2-4d6bab476d53\" (UID: \"631487f2-a57f-4dd0-91c2-4d6bab476d53\") "
Sep 17 18:30:09.647720 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.647527    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/631487f2-a57f-4dd0-91c2-4d6bab476d53-tmp-dir\") pod \"631487f2-a57f-4dd0-91c2-4d6bab476d53\" (UID: \"631487f2-a57f-4dd0-91c2-4d6bab476d53\") "
Sep 17 18:30:09.647720 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.647551    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/631487f2-a57f-4dd0-91c2-4d6bab476d53-cabundle-cert\") pod \"631487f2-a57f-4dd0-91c2-4d6bab476d53\" (UID: \"631487f2-a57f-4dd0-91c2-4d6bab476d53\") "
Sep 17 18:30:09.647720 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.647593    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/631487f2-a57f-4dd0-91c2-4d6bab476d53-dshm\") pod \"631487f2-a57f-4dd0-91c2-4d6bab476d53\" (UID: \"631487f2-a57f-4dd0-91c2-4d6bab476d53\") "
Sep 17 18:30:09.647720 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.647615    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/631487f2-a57f-4dd0-91c2-4d6bab476d53-tls-certs\") pod \"631487f2-a57f-4dd0-91c2-4d6bab476d53\" (UID: \"631487f2-a57f-4dd0-91c2-4d6bab476d53\") "
Sep 17 18:30:09.647720 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.647637    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/631487f2-a57f-4dd0-91c2-4d6bab476d53-home\") pod \"631487f2-a57f-4dd0-91c2-4d6bab476d53\" (UID: \"631487f2-a57f-4dd0-91c2-4d6bab476d53\") "
Sep 17 18:30:09.648072 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.647767    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631487f2-a57f-4dd0-91c2-4d6bab476d53-model-cache" (OuterVolumeSpecName: "model-cache") pod "631487f2-a57f-4dd0-91c2-4d6bab476d53" (UID: "631487f2-a57f-4dd0-91c2-4d6bab476d53"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:09.648072 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.647898    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/631487f2-a57f-4dd0-91c2-4d6bab476d53-model-cache\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:30:09.648072 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.648036    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/631487f2-a57f-4dd0-91c2-4d6bab476d53-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "631487f2-a57f-4dd0-91c2-4d6bab476d53" (UID: "631487f2-a57f-4dd0-91c2-4d6bab476d53"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:30:09.648449 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.648423    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631487f2-a57f-4dd0-91c2-4d6bab476d53-home" (OuterVolumeSpecName: "home") pod "631487f2-a57f-4dd0-91c2-4d6bab476d53" (UID: "631487f2-a57f-4dd0-91c2-4d6bab476d53"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:09.650101 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.650075    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/631487f2-a57f-4dd0-91c2-4d6bab476d53-kube-api-access-bxs6x" (OuterVolumeSpecName: "kube-api-access-bxs6x") pod "631487f2-a57f-4dd0-91c2-4d6bab476d53" (UID: "631487f2-a57f-4dd0-91c2-4d6bab476d53"). InnerVolumeSpecName "kube-api-access-bxs6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:30:09.650210 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.650070    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/631487f2-a57f-4dd0-91c2-4d6bab476d53-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "631487f2-a57f-4dd0-91c2-4d6bab476d53" (UID: "631487f2-a57f-4dd0-91c2-4d6bab476d53"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:30:09.650503 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.650486    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631487f2-a57f-4dd0-91c2-4d6bab476d53-dshm" (OuterVolumeSpecName: "dshm") pod "631487f2-a57f-4dd0-91c2-4d6bab476d53" (UID: "631487f2-a57f-4dd0-91c2-4d6bab476d53"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:09.661048 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.661019    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631487f2-a57f-4dd0-91c2-4d6bab476d53-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "631487f2-a57f-4dd0-91c2-4d6bab476d53" (UID: "631487f2-a57f-4dd0-91c2-4d6bab476d53"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:09.669136 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.669109    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631487f2-a57f-4dd0-91c2-4d6bab476d53-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "631487f2-a57f-4dd0-91c2-4d6bab476d53" (UID: "631487f2-a57f-4dd0-91c2-4d6bab476d53"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:09.694522 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.694498    2583 generic.go:356] "Generic (PLEG): container finished" podID="631487f2-a57f-4dd0-91c2-4d6bab476d53" containerID="d7cff33f2af4e6d53d771172ad7aa5be516f1605d93f026816f0fc05f5f11722" exitCode=0
Sep 17 18:30:09.694627 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.694548    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"
Sep 17 18:30:09.694687 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.694548    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x" event={"ID":"631487f2-a57f-4dd0-91c2-4d6bab476d53","Type":"ContainerDied","Data":"d7cff33f2af4e6d53d771172ad7aa5be516f1605d93f026816f0fc05f5f11722"}
Sep 17 18:30:09.694730 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.694697    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x" event={"ID":"631487f2-a57f-4dd0-91c2-4d6bab476d53","Type":"ContainerDied","Data":"dbdd3c42a610600d592cfff9d1de00c1ef854c8ce57a90868c5b1f4c6662b65c"}
Sep 17 18:30:09.694730 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.694726    2583 scope.go:117] "RemoveContainer" containerID="d7cff33f2af4e6d53d771172ad7aa5be516f1605d93f026816f0fc05f5f11722"
Sep 17 18:30:09.703097 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.703078    2583 scope.go:117] "RemoveContainer" containerID="850a6329e0a180d82fd60f4d7d5d3a30cdfbff34ede63718208fe978451bd0fb"
Sep 17 18:30:09.710189 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.710175    2583 scope.go:117] "RemoveContainer" containerID="d7cff33f2af4e6d53d771172ad7aa5be516f1605d93f026816f0fc05f5f11722"
Sep 17 18:30:09.710480 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:30:09.710457    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7cff33f2af4e6d53d771172ad7aa5be516f1605d93f026816f0fc05f5f11722\": container with ID starting with d7cff33f2af4e6d53d771172ad7aa5be516f1605d93f026816f0fc05f5f11722 not found: ID does not exist" containerID="d7cff33f2af4e6d53d771172ad7aa5be516f1605d93f026816f0fc05f5f11722"
Sep 17 18:30:09.710565 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.710484    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7cff33f2af4e6d53d771172ad7aa5be516f1605d93f026816f0fc05f5f11722"} err="failed to get container status \"d7cff33f2af4e6d53d771172ad7aa5be516f1605d93f026816f0fc05f5f11722\": rpc error: code = NotFound desc = could not find container \"d7cff33f2af4e6d53d771172ad7aa5be516f1605d93f026816f0fc05f5f11722\": container with ID starting with d7cff33f2af4e6d53d771172ad7aa5be516f1605d93f026816f0fc05f5f11722 not found: ID does not exist"
Sep 17 18:30:09.710565 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.710501    2583 scope.go:117] "RemoveContainer" containerID="850a6329e0a180d82fd60f4d7d5d3a30cdfbff34ede63718208fe978451bd0fb"
Sep 17 18:30:09.710801 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:30:09.710781    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"850a6329e0a180d82fd60f4d7d5d3a30cdfbff34ede63718208fe978451bd0fb\": container with ID starting with 850a6329e0a180d82fd60f4d7d5d3a30cdfbff34ede63718208fe978451bd0fb not found: ID does not exist" containerID="850a6329e0a180d82fd60f4d7d5d3a30cdfbff34ede63718208fe978451bd0fb"
Sep 17 18:30:09.710854 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.710806    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"850a6329e0a180d82fd60f4d7d5d3a30cdfbff34ede63718208fe978451bd0fb"} err="failed to get container status \"850a6329e0a180d82fd60f4d7d5d3a30cdfbff34ede63718208fe978451bd0fb\": rpc error: code = NotFound desc = could not find container \"850a6329e0a180d82fd60f4d7d5d3a30cdfbff34ede63718208fe978451bd0fb\": container with ID starting with 850a6329e0a180d82fd60f4d7d5d3a30cdfbff34ede63718208fe978451bd0fb not found: ID does not exist"
Sep 17 18:30:09.717159 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.717137    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"]
Sep 17 18:30:09.722996 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.722977    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5c55p7x"]
Sep 17 18:30:09.748683 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.748645    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxs6x\" (UniqueName: \"kubernetes.io/projected/631487f2-a57f-4dd0-91c2-4d6bab476d53-kube-api-access-bxs6x\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:30:09.748683 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.748684    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/631487f2-a57f-4dd0-91c2-4d6bab476d53-kserve-provision-location\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:30:09.748813 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.748695    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/631487f2-a57f-4dd0-91c2-4d6bab476d53-tmp-dir\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:30:09.748813 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.748703    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/631487f2-a57f-4dd0-91c2-4d6bab476d53-cabundle-cert\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:30:09.748813 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.748712    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/631487f2-a57f-4dd0-91c2-4d6bab476d53-dshm\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:30:09.748813 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.748720    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/631487f2-a57f-4dd0-91c2-4d6bab476d53-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:30:09.748813 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:09.748727    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/631487f2-a57f-4dd0-91c2-4d6bab476d53-home\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:30:11.039552 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:11.039516    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="631487f2-a57f-4dd0-91c2-4d6bab476d53" path="/var/lib/kubelet/pods/631487f2-a57f-4dd0-91c2-4d6bab476d53/volumes"
Sep 17 18:30:11.843313 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:11.843270    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn" podUID="64ac746a-f9f0-4e75-94a9-94ba5d0941af" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:30:21.843328 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:21.843280    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn" podUID="64ac746a-f9f0-4e75-94a9-94ba5d0941af" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:30:26.229491 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:26.229434    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn"
Sep 17 18:30:26.383357 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:26.383330    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64ac746a-f9f0-4e75-94a9-94ba5d0941af-tls-certs\") pod \"64ac746a-f9f0-4e75-94a9-94ba5d0941af\" (UID: \"64ac746a-f9f0-4e75-94a9-94ba5d0941af\") "
Sep 17 18:30:26.383518 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:26.383383    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s68jw\" (UniqueName: \"kubernetes.io/projected/64ac746a-f9f0-4e75-94a9-94ba5d0941af-kube-api-access-s68jw\") pod \"64ac746a-f9f0-4e75-94a9-94ba5d0941af\" (UID: \"64ac746a-f9f0-4e75-94a9-94ba5d0941af\") "
Sep 17 18:30:26.385350 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:26.385321    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64ac746a-f9f0-4e75-94a9-94ba5d0941af-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "64ac746a-f9f0-4e75-94a9-94ba5d0941af" (UID: "64ac746a-f9f0-4e75-94a9-94ba5d0941af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:30:26.385449 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:26.385380    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ac746a-f9f0-4e75-94a9-94ba5d0941af-kube-api-access-s68jw" (OuterVolumeSpecName: "kube-api-access-s68jw") pod "64ac746a-f9f0-4e75-94a9-94ba5d0941af" (UID: "64ac746a-f9f0-4e75-94a9-94ba5d0941af"). InnerVolumeSpecName "kube-api-access-s68jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:30:26.484519 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:26.484450    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64ac746a-f9f0-4e75-94a9-94ba5d0941af-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:30:26.484519 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:26.484476    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s68jw\" (UniqueName: \"kubernetes.io/projected/64ac746a-f9f0-4e75-94a9-94ba5d0941af-kube-api-access-s68jw\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:30:26.771443 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:26.771353    2583 generic.go:356] "Generic (PLEG): container finished" podID="64ac746a-f9f0-4e75-94a9-94ba5d0941af" containerID="a29213e52eefee0a69abd1b0423fa169626b16ebd37c851da131eb8af8555d72" exitCode=0
Sep 17 18:30:26.771443 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:26.771395    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn"
Sep 17 18:30:26.771712 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:26.771445    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn" event={"ID":"64ac746a-f9f0-4e75-94a9-94ba5d0941af","Type":"ContainerDied","Data":"a29213e52eefee0a69abd1b0423fa169626b16ebd37c851da131eb8af8555d72"}
Sep 17 18:30:26.771712 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:26.771505    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn" event={"ID":"64ac746a-f9f0-4e75-94a9-94ba5d0941af","Type":"ContainerDied","Data":"bcc76e78cc14050662345f129d928f6cd5ce045e6837a25182edcc80b6474b48"}
Sep 17 18:30:26.771712 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:26.771536    2583 scope.go:117] "RemoveContainer" containerID="a29213e52eefee0a69abd1b0423fa169626b16ebd37c851da131eb8af8555d72"
Sep 17 18:30:26.779996 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:26.779979    2583 scope.go:117] "RemoveContainer" containerID="a29213e52eefee0a69abd1b0423fa169626b16ebd37c851da131eb8af8555d72"
Sep 17 18:30:26.780272 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:30:26.780253    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a29213e52eefee0a69abd1b0423fa169626b16ebd37c851da131eb8af8555d72\": container with ID starting with a29213e52eefee0a69abd1b0423fa169626b16ebd37c851da131eb8af8555d72 not found: ID does not exist" containerID="a29213e52eefee0a69abd1b0423fa169626b16ebd37c851da131eb8af8555d72"
Sep 17 18:30:26.780331 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:26.780279    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a29213e52eefee0a69abd1b0423fa169626b16ebd37c851da131eb8af8555d72"} err="failed to get container status \"a29213e52eefee0a69abd1b0423fa169626b16ebd37c851da131eb8af8555d72\": rpc error: code = NotFound desc = could not find container \"a29213e52eefee0a69abd1b0423fa169626b16ebd37c851da131eb8af8555d72\": container with ID starting with a29213e52eefee0a69abd1b0423fa169626b16ebd37c851da131eb8af8555d72 not found: ID does not exist"
Sep 17 18:30:26.791584 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:26.791561    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn"]
Sep 17 18:30:26.797349 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:26.797329    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-nflgn"]
Sep 17 18:30:27.040243 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:27.040167    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64ac746a-f9f0-4e75-94a9-94ba5d0941af" path="/var/lib/kubelet/pods/64ac746a-f9f0-4e75-94a9-94ba5d0941af/volumes"
Sep 17 18:30:43.617583 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.617550    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"]
Sep 17 18:30:43.617995 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.617860    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64ac746a-f9f0-4e75-94a9-94ba5d0941af" containerName="main"
Sep 17 18:30:43.617995 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.617872    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="64ac746a-f9f0-4e75-94a9-94ba5d0941af" containerName="main"
Sep 17 18:30:43.617995 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.617883    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="202e8856-86a5-4a0a-9462-7b7dd3f2b767" containerName="main"
Sep 17 18:30:43.617995 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.617889    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="202e8856-86a5-4a0a-9462-7b7dd3f2b767" containerName="main"
Sep 17 18:30:43.617995 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.617896    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="631487f2-a57f-4dd0-91c2-4d6bab476d53" containerName="main"
Sep 17 18:30:43.617995 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.617903    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="631487f2-a57f-4dd0-91c2-4d6bab476d53" containerName="main"
Sep 17 18:30:43.617995 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.617910    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="631487f2-a57f-4dd0-91c2-4d6bab476d53" containerName="storage-initializer"
Sep 17 18:30:43.617995 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.617916    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="631487f2-a57f-4dd0-91c2-4d6bab476d53" containerName="storage-initializer"
Sep 17 18:30:43.617995 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.617931    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="202e8856-86a5-4a0a-9462-7b7dd3f2b767" containerName="storage-initializer"
Sep 17 18:30:43.617995 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.617937    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="202e8856-86a5-4a0a-9462-7b7dd3f2b767" containerName="storage-initializer"
Sep 17 18:30:43.617995 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.617980    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="64ac746a-f9f0-4e75-94a9-94ba5d0941af" containerName="main"
Sep 17 18:30:43.617995 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.617987    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="202e8856-86a5-4a0a-9462-7b7dd3f2b767" containerName="main"
Sep 17 18:30:43.617995 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.617992    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="631487f2-a57f-4dd0-91c2-4d6bab476d53" containerName="main"
Sep 17 18:30:43.620467 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.620450    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"
Sep 17 18:30:43.623330 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.623302    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:30:43.623330 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.623327    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Sep 17 18:30:43.624027 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.624012    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Sep 17 18:30:43.628030 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.628007    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Sep 17 18:30:43.628253 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.628234    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Sep 17 18:30:43.629137 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.629118    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-5j4mz\""
Sep 17 18:30:43.636386 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.636366    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"]
Sep 17 18:30:43.705299 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.705271    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55f45595-b912-4e33-8658-1c31b91dc07d-dshm\") pod \"lora-single-adapter-test-kserve-69f6bfb88b-vmvkh\" (UID: \"55f45595-b912-4e33-8658-1c31b91dc07d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"
Sep 17 18:30:43.705456 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.705304    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55f45595-b912-4e33-8658-1c31b91dc07d-tmp-dir\") pod \"lora-single-adapter-test-kserve-69f6bfb88b-vmvkh\" (UID: \"55f45595-b912-4e33-8658-1c31b91dc07d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"
Sep 17 18:30:43.705456 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.705341    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55f45595-b912-4e33-8658-1c31b91dc07d-cabundle-cert\") pod \"lora-single-adapter-test-kserve-69f6bfb88b-vmvkh\" (UID: \"55f45595-b912-4e33-8658-1c31b91dc07d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"
Sep 17 18:30:43.705456 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.705365    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g2lz\" (UniqueName: \"kubernetes.io/projected/55f45595-b912-4e33-8658-1c31b91dc07d-kube-api-access-9g2lz\") pod \"lora-single-adapter-test-kserve-69f6bfb88b-vmvkh\" (UID: \"55f45595-b912-4e33-8658-1c31b91dc07d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"
Sep 17 18:30:43.705456 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.705389    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55f45595-b912-4e33-8658-1c31b91dc07d-tls-certs\") pod \"lora-single-adapter-test-kserve-69f6bfb88b-vmvkh\" (UID: \"55f45595-b912-4e33-8658-1c31b91dc07d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"
Sep 17 18:30:43.705456 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.705405    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55f45595-b912-4e33-8658-1c31b91dc07d-home\") pod \"lora-single-adapter-test-kserve-69f6bfb88b-vmvkh\" (UID: \"55f45595-b912-4e33-8658-1c31b91dc07d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"
Sep 17 18:30:43.705456 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.705423    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55f45595-b912-4e33-8658-1c31b91dc07d-model-cache\") pod \"lora-single-adapter-test-kserve-69f6bfb88b-vmvkh\" (UID: \"55f45595-b912-4e33-8658-1c31b91dc07d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"
Sep 17 18:30:43.705456 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.705446    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55f45595-b912-4e33-8658-1c31b91dc07d-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-69f6bfb88b-vmvkh\" (UID: \"55f45595-b912-4e33-8658-1c31b91dc07d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"
Sep 17 18:30:43.806302 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.806267    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55f45595-b912-4e33-8658-1c31b91dc07d-dshm\") pod \"lora-single-adapter-test-kserve-69f6bfb88b-vmvkh\" (UID: \"55f45595-b912-4e33-8658-1c31b91dc07d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"
Sep 17 18:30:43.806302 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.806305    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55f45595-b912-4e33-8658-1c31b91dc07d-tmp-dir\") pod \"lora-single-adapter-test-kserve-69f6bfb88b-vmvkh\" (UID: \"55f45595-b912-4e33-8658-1c31b91dc07d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"
Sep 17 18:30:43.806533 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.806330    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55f45595-b912-4e33-8658-1c31b91dc07d-cabundle-cert\") pod \"lora-single-adapter-test-kserve-69f6bfb88b-vmvkh\" (UID: \"55f45595-b912-4e33-8658-1c31b91dc07d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"
Sep 17 18:30:43.806533 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.806348    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g2lz\" (UniqueName: \"kubernetes.io/projected/55f45595-b912-4e33-8658-1c31b91dc07d-kube-api-access-9g2lz\") pod \"lora-single-adapter-test-kserve-69f6bfb88b-vmvkh\" (UID: \"55f45595-b912-4e33-8658-1c31b91dc07d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"
Sep 17 18:30:43.806533 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.806368    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55f45595-b912-4e33-8658-1c31b91dc07d-tls-certs\") pod \"lora-single-adapter-test-kserve-69f6bfb88b-vmvkh\" (UID: \"55f45595-b912-4e33-8658-1c31b91dc07d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"
Sep 17 18:30:43.806533 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.806383    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55f45595-b912-4e33-8658-1c31b91dc07d-home\") pod \"lora-single-adapter-test-kserve-69f6bfb88b-vmvkh\" (UID: \"55f45595-b912-4e33-8658-1c31b91dc07d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"
Sep 17 18:30:43.806533 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.806478    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55f45595-b912-4e33-8658-1c31b91dc07d-model-cache\") pod \"lora-single-adapter-test-kserve-69f6bfb88b-vmvkh\" (UID: \"55f45595-b912-4e33-8658-1c31b91dc07d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"
Sep 17 18:30:43.806533 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.806503    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55f45595-b912-4e33-8658-1c31b91dc07d-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-69f6bfb88b-vmvkh\" (UID: \"55f45595-b912-4e33-8658-1c31b91dc07d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"
Sep 17 18:30:43.806883 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.806777    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55f45595-b912-4e33-8658-1c31b91dc07d-tmp-dir\") pod \"lora-single-adapter-test-kserve-69f6bfb88b-vmvkh\" (UID: \"55f45595-b912-4e33-8658-1c31b91dc07d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"
Sep 17 18:30:43.806940 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.806919    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55f45595-b912-4e33-8658-1c31b91dc07d-home\") pod \"lora-single-adapter-test-kserve-69f6bfb88b-vmvkh\" (UID: \"55f45595-b912-4e33-8658-1c31b91dc07d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"
Sep 17 18:30:43.806986 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.806938    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55f45595-b912-4e33-8658-1c31b91dc07d-model-cache\") pod \"lora-single-adapter-test-kserve-69f6bfb88b-vmvkh\" (UID: \"55f45595-b912-4e33-8658-1c31b91dc07d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"
Sep 17 18:30:43.807184 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.807164    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55f45595-b912-4e33-8658-1c31b91dc07d-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-69f6bfb88b-vmvkh\" (UID: \"55f45595-b912-4e33-8658-1c31b91dc07d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"
Sep 17 18:30:43.807244 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.807174    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55f45595-b912-4e33-8658-1c31b91dc07d-cabundle-cert\") pod \"lora-single-adapter-test-kserve-69f6bfb88b-vmvkh\" (UID: \"55f45595-b912-4e33-8658-1c31b91dc07d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"
Sep 17 18:30:43.808754 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.808731    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55f45595-b912-4e33-8658-1c31b91dc07d-dshm\") pod \"lora-single-adapter-test-kserve-69f6bfb88b-vmvkh\" (UID: \"55f45595-b912-4e33-8658-1c31b91dc07d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"
Sep 17 18:30:43.808966 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.808944    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55f45595-b912-4e33-8658-1c31b91dc07d-tls-certs\") pod \"lora-single-adapter-test-kserve-69f6bfb88b-vmvkh\" (UID: \"55f45595-b912-4e33-8658-1c31b91dc07d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"
Sep 17 18:30:43.815539 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.815515    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g2lz\" (UniqueName: \"kubernetes.io/projected/55f45595-b912-4e33-8658-1c31b91dc07d-kube-api-access-9g2lz\") pod \"lora-single-adapter-test-kserve-69f6bfb88b-vmvkh\" (UID: \"55f45595-b912-4e33-8658-1c31b91dc07d\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"
Sep 17 18:30:43.930923 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:43.930854    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"
Sep 17 18:30:44.213531 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:44.213452    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"]
Sep 17 18:30:44.216754 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:30:44.216723    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f45595_b912_4e33_8658_1c31b91dc07d.slice/crio-ce1dd7b65e8f6753a15c42bee4c6998f1d8ec5e1cb351a66758f826928032e38 WatchSource:0}: Error finding container ce1dd7b65e8f6753a15c42bee4c6998f1d8ec5e1cb351a66758f826928032e38: Status 404 returned error can't find the container with id ce1dd7b65e8f6753a15c42bee4c6998f1d8ec5e1cb351a66758f826928032e38
Sep 17 18:30:44.846374 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:44.846336    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh" event={"ID":"55f45595-b912-4e33-8658-1c31b91dc07d","Type":"ContainerStarted","Data":"c2aeb3ce7f8513e8b1727d402af8c8441fffe1acfdbf6e9686377e359cbd4fe5"}
Sep 17 18:30:44.846820 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:44.846391    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh" event={"ID":"55f45595-b912-4e33-8658-1c31b91dc07d","Type":"ContainerStarted","Data":"ce1dd7b65e8f6753a15c42bee4c6998f1d8ec5e1cb351a66758f826928032e38"}
Sep 17 18:30:47.859784 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:47.859747    2583 generic.go:356] "Generic (PLEG): container finished" podID="55f45595-b912-4e33-8658-1c31b91dc07d" containerID="c2aeb3ce7f8513e8b1727d402af8c8441fffe1acfdbf6e9686377e359cbd4fe5" exitCode=0
Sep 17 18:30:47.860170 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:47.859821    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh" event={"ID":"55f45595-b912-4e33-8658-1c31b91dc07d","Type":"ContainerDied","Data":"c2aeb3ce7f8513e8b1727d402af8c8441fffe1acfdbf6e9686377e359cbd4fe5"}
Sep 17 18:30:48.866015 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:48.865976    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh" event={"ID":"55f45595-b912-4e33-8658-1c31b91dc07d","Type":"ContainerStarted","Data":"7604f8c49c08713d37dfcac13a6c2012311b4c53d00296e95461d9b26010cb65"}
Sep 17 18:30:48.895634 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:48.895582    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh" podStartSLOduration=3.253271078 podStartE2EDuration="5.895564712s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.642293634s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:48.893854748 +0000 UTC m=+3359.985161524" watchObservedRunningTime="2026-09-17 18:30:48.895564712 +0000 UTC m=+3359.986871477"
Sep 17 18:30:48.931041 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:48.931009    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"
Sep 17 18:30:53.931837 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:53.931799    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"
Sep 17 18:30:53.933613 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:30:53.933583    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh" podUID="55f45595-b912-4e33-8658-1c31b91dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 18:31:03.931344 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:03.931293    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh" podUID="55f45595-b912-4e33-8658-1c31b91dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 18:31:11.267920 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.267886    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"]
Sep 17 18:31:11.271089 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.271067    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:31:11.273569 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.273541    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:31:11.273714 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.273641    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 17 18:31:11.273962 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.273942    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 17 18:31:11.274074 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.274030    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-5c6vs\""
Sep 17 18:31:11.274074 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.274010    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 17 18:31:11.274343 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.274326    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 17 18:31:11.275173 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.275152    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-gcrsm\""
Sep 17 18:31:11.287435 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.287411    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"]
Sep 17 18:31:11.347048 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.347022    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb46d7f3-4d8b-4716-b10f-d104098a0690-home\") pod \"custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2\" (UID: \"fb46d7f3-4d8b-4716-b10f-d104098a0690\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:31:11.347210 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.347058    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb46d7f3-4d8b-4716-b10f-d104098a0690-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2\" (UID: \"fb46d7f3-4d8b-4716-b10f-d104098a0690\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:31:11.347210 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.347084    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb46d7f3-4d8b-4716-b10f-d104098a0690-model-cache\") pod \"custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2\" (UID: \"fb46d7f3-4d8b-4716-b10f-d104098a0690\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:31:11.347210 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.347141    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tmx2\" (UniqueName: \"kubernetes.io/projected/fb46d7f3-4d8b-4716-b10f-d104098a0690-kube-api-access-2tmx2\") pod \"custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2\" (UID: \"fb46d7f3-4d8b-4716-b10f-d104098a0690\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:31:11.347210 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.347170    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb46d7f3-4d8b-4716-b10f-d104098a0690-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2\" (UID: \"fb46d7f3-4d8b-4716-b10f-d104098a0690\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:31:11.347210 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.347189    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb46d7f3-4d8b-4716-b10f-d104098a0690-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2\" (UID: \"fb46d7f3-4d8b-4716-b10f-d104098a0690\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:31:11.347457 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.347314    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb46d7f3-4d8b-4716-b10f-d104098a0690-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2\" (UID: \"fb46d7f3-4d8b-4716-b10f-d104098a0690\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:31:11.347457 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.347357    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb46d7f3-4d8b-4716-b10f-d104098a0690-dshm\") pod \"custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2\" (UID: \"fb46d7f3-4d8b-4716-b10f-d104098a0690\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:31:11.448915 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.448873    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb46d7f3-4d8b-4716-b10f-d104098a0690-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2\" (UID: \"fb46d7f3-4d8b-4716-b10f-d104098a0690\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:31:11.449099 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.448932    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb46d7f3-4d8b-4716-b10f-d104098a0690-dshm\") pod \"custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2\" (UID: \"fb46d7f3-4d8b-4716-b10f-d104098a0690\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:31:11.449099 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.449005    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb46d7f3-4d8b-4716-b10f-d104098a0690-home\") pod \"custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2\" (UID: \"fb46d7f3-4d8b-4716-b10f-d104098a0690\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:31:11.449099 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.449033    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb46d7f3-4d8b-4716-b10f-d104098a0690-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2\" (UID: \"fb46d7f3-4d8b-4716-b10f-d104098a0690\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:31:11.449099 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.449064    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb46d7f3-4d8b-4716-b10f-d104098a0690-model-cache\") pod \"custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2\" (UID: \"fb46d7f3-4d8b-4716-b10f-d104098a0690\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:31:11.449327 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.449098    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tmx2\" (UniqueName: \"kubernetes.io/projected/fb46d7f3-4d8b-4716-b10f-d104098a0690-kube-api-access-2tmx2\") pod \"custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2\" (UID: \"fb46d7f3-4d8b-4716-b10f-d104098a0690\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:31:11.449327 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.449142    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb46d7f3-4d8b-4716-b10f-d104098a0690-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2\" (UID: \"fb46d7f3-4d8b-4716-b10f-d104098a0690\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:31:11.449327 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.449168    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb46d7f3-4d8b-4716-b10f-d104098a0690-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2\" (UID: \"fb46d7f3-4d8b-4716-b10f-d104098a0690\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:31:11.449327 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.449306    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb46d7f3-4d8b-4716-b10f-d104098a0690-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2\" (UID: \"fb46d7f3-4d8b-4716-b10f-d104098a0690\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:31:11.449538 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.449343    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb46d7f3-4d8b-4716-b10f-d104098a0690-model-cache\") pod \"custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2\" (UID: \"fb46d7f3-4d8b-4716-b10f-d104098a0690\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:31:11.449538 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.449425    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb46d7f3-4d8b-4716-b10f-d104098a0690-home\") pod \"custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2\" (UID: \"fb46d7f3-4d8b-4716-b10f-d104098a0690\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:31:11.449680 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.449577    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb46d7f3-4d8b-4716-b10f-d104098a0690-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2\" (UID: \"fb46d7f3-4d8b-4716-b10f-d104098a0690\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:31:11.449965 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.449941    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb46d7f3-4d8b-4716-b10f-d104098a0690-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2\" (UID: \"fb46d7f3-4d8b-4716-b10f-d104098a0690\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:31:11.451294 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.451265    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb46d7f3-4d8b-4716-b10f-d104098a0690-dshm\") pod \"custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2\" (UID: \"fb46d7f3-4d8b-4716-b10f-d104098a0690\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:31:11.451586 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.451568    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb46d7f3-4d8b-4716-b10f-d104098a0690-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2\" (UID: \"fb46d7f3-4d8b-4716-b10f-d104098a0690\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:31:11.457421 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.457400    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tmx2\" (UniqueName: \"kubernetes.io/projected/fb46d7f3-4d8b-4716-b10f-d104098a0690-kube-api-access-2tmx2\") pod \"custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2\" (UID: \"fb46d7f3-4d8b-4716-b10f-d104098a0690\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:31:11.582256 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.582177    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:31:11.866117 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.866078    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"]
Sep 17 18:31:11.868824 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:31:11.868795    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb46d7f3_4d8b_4716_b10f_d104098a0690.slice/crio-ccd105464eeed2bba0096121859f915dfad611370646df5505fdfeac1604aac5 WatchSource:0}: Error finding container ccd105464eeed2bba0096121859f915dfad611370646df5505fdfeac1604aac5: Status 404 returned error can't find the container with id ccd105464eeed2bba0096121859f915dfad611370646df5505fdfeac1604aac5
Sep 17 18:31:11.967815 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.967775    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2" event={"ID":"fb46d7f3-4d8b-4716-b10f-d104098a0690","Type":"ContainerStarted","Data":"fe53438c270ee18a442afa2932f04611a9cb00b801071335e456e0c56ca5d270"}
Sep 17 18:31:11.967963 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.967840    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2" event={"ID":"fb46d7f3-4d8b-4716-b10f-d104098a0690","Type":"ContainerStarted","Data":"ccd105464eeed2bba0096121859f915dfad611370646df5505fdfeac1604aac5"}
Sep 17 18:31:11.967963 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:11.967886    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:31:12.977059 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:12.977017    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2" event={"ID":"fb46d7f3-4d8b-4716-b10f-d104098a0690","Type":"ContainerStarted","Data":"9dacf5b7603b15c907a14f8f32a61b153360c557abd89d06ec917139aedd497f"}
Sep 17 18:31:13.932234 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:13.932194    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh" podUID="55f45595-b912-4e33-8658-1c31b91dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 18:31:14.989178 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:14.989141    2583 generic.go:356] "Generic (PLEG): container finished" podID="fb46d7f3-4d8b-4716-b10f-d104098a0690" containerID="9dacf5b7603b15c907a14f8f32a61b153360c557abd89d06ec917139aedd497f" exitCode=0
Sep 17 18:31:14.989178 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:14.989183    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2" event={"ID":"fb46d7f3-4d8b-4716-b10f-d104098a0690","Type":"ContainerDied","Data":"9dacf5b7603b15c907a14f8f32a61b153360c557abd89d06ec917139aedd497f"}
Sep 17 18:31:15.995976 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:15.995935    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2" event={"ID":"fb46d7f3-4d8b-4716-b10f-d104098a0690","Type":"ContainerStarted","Data":"1a0f1d30f17454db8ecdbd2b3c9aa805d352e90675c7f7b293ebe5248659d42d"}
Sep 17 18:31:16.026310 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:16.026250    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2" podStartSLOduration=2.755730432 podStartE2EDuration="5.026231177s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.270500745s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:31:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:31:16.024364863 +0000 UTC m=+3387.115671628" watchObservedRunningTime="2026-09-17 18:31:16.026231177 +0000 UTC m=+3387.117537943"
Sep 17 18:31:21.582679 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:21.582637    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:31:21.583180 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:21.582722    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:31:21.584209 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:21.584175    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2" podUID="fb46d7f3-4d8b-4716-b10f-d104098a0690" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8001/health\": dial tcp 10.134.0.54:8001: connect: connection refused"
Sep 17 18:31:21.601208 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:21.601184    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:31:23.931315 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:23.931253    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh" podUID="55f45595-b912-4e33-8658-1c31b91dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 18:31:31.583673 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:31.583609    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2" podUID="fb46d7f3-4d8b-4716-b10f-d104098a0690" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8001/health\": dial tcp 10.134.0.54:8001: connect: connection refused"
Sep 17 18:31:33.931849 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:33.931791    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh" podUID="55f45595-b912-4e33-8658-1c31b91dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 18:31:41.582973 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:41.582926    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2" podUID="fb46d7f3-4d8b-4716-b10f-d104098a0690" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8001/health\": dial tcp 10.134.0.54:8001: connect: connection refused"
Sep 17 18:31:43.931690 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:43.931623    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh" podUID="55f45595-b912-4e33-8658-1c31b91dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 18:31:51.583037 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:51.582971    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2" podUID="fb46d7f3-4d8b-4716-b10f-d104098a0690" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8001/health\": dial tcp 10.134.0.54:8001: connect: connection refused"
Sep 17 18:31:53.931457 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:31:53.931404    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh" podUID="55f45595-b912-4e33-8658-1c31b91dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 17 18:32:01.583007 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:01.582943    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2" podUID="fb46d7f3-4d8b-4716-b10f-d104098a0690" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8001/health\": dial tcp 10.134.0.54:8001: connect: connection refused"
Sep 17 18:32:03.941593 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:03.941563    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"
Sep 17 18:32:03.949802 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:03.949776    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"
Sep 17 18:32:11.583233 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:11.583183    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2" podUID="fb46d7f3-4d8b-4716-b10f-d104098a0690" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8001/health\": dial tcp 10.134.0.54:8001: connect: connection refused"
Sep 17 18:32:21.582889 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:21.582849    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2" podUID="fb46d7f3-4d8b-4716-b10f-d104098a0690" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8001/health\": dial tcp 10.134.0.54:8001: connect: connection refused"
Sep 17 18:32:28.838235 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:28.838184    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-69f6bfb88b-vmvkh_55f45595-b912-4e33-8658-1c31b91dc07d/storage-initializer/0.log"
Sep 17 18:32:28.859273 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:28.859245    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-69f6bfb88b-vmvkh_55f45595-b912-4e33-8658-1c31b91dc07d/main/0.log"
Sep 17 18:32:30.155358 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:30.155325    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"]
Sep 17 18:32:31.583178 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:31.583124    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2" podUID="fb46d7f3-4d8b-4716-b10f-d104098a0690" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8001/health\": dial tcp 10.134.0.54:8001: connect: connection refused"
Sep 17 18:32:41.583365 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:41.583311    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2" podUID="fb46d7f3-4d8b-4716-b10f-d104098a0690" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8001/health\": dial tcp 10.134.0.54:8001: connect: connection refused"
Sep 17 18:32:45.177264 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:45.177195    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh" podUID="55f45595-b912-4e33-8658-1c31b91dc07d" containerName="main" containerID="cri-o://7604f8c49c08713d37dfcac13a6c2012311b4c53d00296e95461d9b26010cb65" gracePeriod=45
Sep 17 18:32:46.038353 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.038330    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"
Sep 17 18:32:46.094066 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.094040    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55f45595-b912-4e33-8658-1c31b91dc07d-tls-certs\") pod \"55f45595-b912-4e33-8658-1c31b91dc07d\" (UID: \"55f45595-b912-4e33-8658-1c31b91dc07d\") "
Sep 17 18:32:46.094258 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.094077    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55f45595-b912-4e33-8658-1c31b91dc07d-cabundle-cert\") pod \"55f45595-b912-4e33-8658-1c31b91dc07d\" (UID: \"55f45595-b912-4e33-8658-1c31b91dc07d\") "
Sep 17 18:32:46.094258 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.094107    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55f45595-b912-4e33-8658-1c31b91dc07d-tmp-dir\") pod \"55f45595-b912-4e33-8658-1c31b91dc07d\" (UID: \"55f45595-b912-4e33-8658-1c31b91dc07d\") "
Sep 17 18:32:46.094258 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.094124    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55f45595-b912-4e33-8658-1c31b91dc07d-home\") pod \"55f45595-b912-4e33-8658-1c31b91dc07d\" (UID: \"55f45595-b912-4e33-8658-1c31b91dc07d\") "
Sep 17 18:32:46.094435 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.094257    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55f45595-b912-4e33-8658-1c31b91dc07d-model-cache\") pod \"55f45595-b912-4e33-8658-1c31b91dc07d\" (UID: \"55f45595-b912-4e33-8658-1c31b91dc07d\") "
Sep 17 18:32:46.094435 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.094290    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9g2lz\" (UniqueName: \"kubernetes.io/projected/55f45595-b912-4e33-8658-1c31b91dc07d-kube-api-access-9g2lz\") pod \"55f45595-b912-4e33-8658-1c31b91dc07d\" (UID: \"55f45595-b912-4e33-8658-1c31b91dc07d\") "
Sep 17 18:32:46.094435 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.094358    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55f45595-b912-4e33-8658-1c31b91dc07d-dshm\") pod \"55f45595-b912-4e33-8658-1c31b91dc07d\" (UID: \"55f45595-b912-4e33-8658-1c31b91dc07d\") "
Sep 17 18:32:46.094435 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.094386    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55f45595-b912-4e33-8658-1c31b91dc07d-kserve-provision-location\") pod \"55f45595-b912-4e33-8658-1c31b91dc07d\" (UID: \"55f45595-b912-4e33-8658-1c31b91dc07d\") "
Sep 17 18:32:46.094677 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.094531    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f45595-b912-4e33-8658-1c31b91dc07d-model-cache" (OuterVolumeSpecName: "model-cache") pod "55f45595-b912-4e33-8658-1c31b91dc07d" (UID: "55f45595-b912-4e33-8658-1c31b91dc07d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:32:46.094677 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.094556    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f45595-b912-4e33-8658-1c31b91dc07d-home" (OuterVolumeSpecName: "home") pod "55f45595-b912-4e33-8658-1c31b91dc07d" (UID: "55f45595-b912-4e33-8658-1c31b91dc07d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:32:46.094677 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.094554    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55f45595-b912-4e33-8658-1c31b91dc07d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "55f45595-b912-4e33-8658-1c31b91dc07d" (UID: "55f45595-b912-4e33-8658-1c31b91dc07d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:32:46.094838 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.094693    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55f45595-b912-4e33-8658-1c31b91dc07d-model-cache\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:32:46.094838 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.094713    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55f45595-b912-4e33-8658-1c31b91dc07d-cabundle-cert\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:32:46.094838 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.094758    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55f45595-b912-4e33-8658-1c31b91dc07d-home\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:32:46.096779 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.096746    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55f45595-b912-4e33-8658-1c31b91dc07d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55f45595-b912-4e33-8658-1c31b91dc07d" (UID: "55f45595-b912-4e33-8658-1c31b91dc07d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:32:46.097129 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.097102    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f45595-b912-4e33-8658-1c31b91dc07d-dshm" (OuterVolumeSpecName: "dshm") pod "55f45595-b912-4e33-8658-1c31b91dc07d" (UID: "55f45595-b912-4e33-8658-1c31b91dc07d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:32:46.097229 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.097125    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f45595-b912-4e33-8658-1c31b91dc07d-kube-api-access-9g2lz" (OuterVolumeSpecName: "kube-api-access-9g2lz") pod "55f45595-b912-4e33-8658-1c31b91dc07d" (UID: "55f45595-b912-4e33-8658-1c31b91dc07d"). InnerVolumeSpecName "kube-api-access-9g2lz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:32:46.112422 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.112363    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f45595-b912-4e33-8658-1c31b91dc07d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "55f45595-b912-4e33-8658-1c31b91dc07d" (UID: "55f45595-b912-4e33-8658-1c31b91dc07d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:32:46.119176 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.119145    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f45595-b912-4e33-8658-1c31b91dc07d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "55f45595-b912-4e33-8658-1c31b91dc07d" (UID: "55f45595-b912-4e33-8658-1c31b91dc07d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:32:46.195092 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.195066    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55f45595-b912-4e33-8658-1c31b91dc07d-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:32:46.195092 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.195091    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55f45595-b912-4e33-8658-1c31b91dc07d-tmp-dir\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:32:46.195465 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.195106    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g2lz\" (UniqueName: \"kubernetes.io/projected/55f45595-b912-4e33-8658-1c31b91dc07d-kube-api-access-9g2lz\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:32:46.195465 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.195117    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55f45595-b912-4e33-8658-1c31b91dc07d-dshm\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:32:46.195465 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.195132    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55f45595-b912-4e33-8658-1c31b91dc07d-kserve-provision-location\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:32:46.441063 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.440984    2583 generic.go:356] "Generic (PLEG): container finished" podID="55f45595-b912-4e33-8658-1c31b91dc07d" containerID="7604f8c49c08713d37dfcac13a6c2012311b4c53d00296e95461d9b26010cb65" exitCode=0
Sep 17 18:32:46.441063 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.441052    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"
Sep 17 18:32:46.441243 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.441065    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh" event={"ID":"55f45595-b912-4e33-8658-1c31b91dc07d","Type":"ContainerDied","Data":"7604f8c49c08713d37dfcac13a6c2012311b4c53d00296e95461d9b26010cb65"}
Sep 17 18:32:46.441243 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.441118    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh" event={"ID":"55f45595-b912-4e33-8658-1c31b91dc07d","Type":"ContainerDied","Data":"ce1dd7b65e8f6753a15c42bee4c6998f1d8ec5e1cb351a66758f826928032e38"}
Sep 17 18:32:46.441243 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.441141    2583 scope.go:117] "RemoveContainer" containerID="7604f8c49c08713d37dfcac13a6c2012311b4c53d00296e95461d9b26010cb65"
Sep 17 18:32:46.449561 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.449544    2583 scope.go:117] "RemoveContainer" containerID="c2aeb3ce7f8513e8b1727d402af8c8441fffe1acfdbf6e9686377e359cbd4fe5"
Sep 17 18:32:46.458030 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.458009    2583 scope.go:117] "RemoveContainer" containerID="7604f8c49c08713d37dfcac13a6c2012311b4c53d00296e95461d9b26010cb65"
Sep 17 18:32:46.458326 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:32:46.458304    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7604f8c49c08713d37dfcac13a6c2012311b4c53d00296e95461d9b26010cb65\": container with ID starting with 7604f8c49c08713d37dfcac13a6c2012311b4c53d00296e95461d9b26010cb65 not found: ID does not exist" containerID="7604f8c49c08713d37dfcac13a6c2012311b4c53d00296e95461d9b26010cb65"
Sep 17 18:32:46.458414 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.458330    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7604f8c49c08713d37dfcac13a6c2012311b4c53d00296e95461d9b26010cb65"} err="failed to get container status \"7604f8c49c08713d37dfcac13a6c2012311b4c53d00296e95461d9b26010cb65\": rpc error: code = NotFound desc = could not find container \"7604f8c49c08713d37dfcac13a6c2012311b4c53d00296e95461d9b26010cb65\": container with ID starting with 7604f8c49c08713d37dfcac13a6c2012311b4c53d00296e95461d9b26010cb65 not found: ID does not exist"
Sep 17 18:32:46.458414 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.458349    2583 scope.go:117] "RemoveContainer" containerID="c2aeb3ce7f8513e8b1727d402af8c8441fffe1acfdbf6e9686377e359cbd4fe5"
Sep 17 18:32:46.458607 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:32:46.458582    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2aeb3ce7f8513e8b1727d402af8c8441fffe1acfdbf6e9686377e359cbd4fe5\": container with ID starting with c2aeb3ce7f8513e8b1727d402af8c8441fffe1acfdbf6e9686377e359cbd4fe5 not found: ID does not exist" containerID="c2aeb3ce7f8513e8b1727d402af8c8441fffe1acfdbf6e9686377e359cbd4fe5"
Sep 17 18:32:46.458652 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.458618    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2aeb3ce7f8513e8b1727d402af8c8441fffe1acfdbf6e9686377e359cbd4fe5"} err="failed to get container status \"c2aeb3ce7f8513e8b1727d402af8c8441fffe1acfdbf6e9686377e359cbd4fe5\": rpc error: code = NotFound desc = could not find container \"c2aeb3ce7f8513e8b1727d402af8c8441fffe1acfdbf6e9686377e359cbd4fe5\": container with ID starting with c2aeb3ce7f8513e8b1727d402af8c8441fffe1acfdbf6e9686377e359cbd4fe5 not found: ID does not exist"
Sep 17 18:32:46.466893 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.466873    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"]
Sep 17 18:32:46.471305 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.471287    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh"]
Sep 17 18:32:46.942613 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:46.942561    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-69f6bfb88b-vmvkh" podUID="55f45595-b912-4e33-8658-1c31b91dc07d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 18:32:47.040571 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:47.040536    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55f45595-b912-4e33-8658-1c31b91dc07d" path="/var/lib/kubelet/pods/55f45595-b912-4e33-8658-1c31b91dc07d/volumes"
Sep 17 18:32:51.583047 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:51.582995    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2" podUID="fb46d7f3-4d8b-4716-b10f-d104098a0690" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8001/health\": dial tcp 10.134.0.54:8001: connect: connection refused"
Sep 17 18:32:59.124468 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.124434    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"]
Sep 17 18:32:59.124881 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.124760    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55f45595-b912-4e33-8658-1c31b91dc07d" containerName="storage-initializer"
Sep 17 18:32:59.124881 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.124771    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="55f45595-b912-4e33-8658-1c31b91dc07d" containerName="storage-initializer"
Sep 17 18:32:59.124881 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.124782    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55f45595-b912-4e33-8658-1c31b91dc07d" containerName="main"
Sep 17 18:32:59.124881 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.124788    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="55f45595-b912-4e33-8658-1c31b91dc07d" containerName="main"
Sep 17 18:32:59.124881 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.124836    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="55f45595-b912-4e33-8658-1c31b91dc07d" containerName="main"
Sep 17 18:32:59.126871 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.126852    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"
Sep 17 18:32:59.130047 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.130015    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Sep 17 18:32:59.130199 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.130049    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:32:59.130199 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.130169    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-tz9pm\""
Sep 17 18:32:59.130440 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.130421    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Sep 17 18:32:59.130522 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.130472    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Sep 17 18:32:59.131047 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.131027    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Sep 17 18:32:59.145365 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.145339    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"]
Sep 17 18:32:59.198058 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.198029    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bde38ba-4188-4608-9c35-9fc65d033eb6-home\") pod \"lora-multiple-adapters-test-kserve-bd5cf78f-g848q\" (UID: \"0bde38ba-4188-4608-9c35-9fc65d033eb6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"
Sep 17 18:32:59.198058 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.198064    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bde38ba-4188-4608-9c35-9fc65d033eb6-dshm\") pod \"lora-multiple-adapters-test-kserve-bd5cf78f-g848q\" (UID: \"0bde38ba-4188-4608-9c35-9fc65d033eb6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"
Sep 17 18:32:59.198293 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.198091    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bde38ba-4188-4608-9c35-9fc65d033eb6-tls-certs\") pod \"lora-multiple-adapters-test-kserve-bd5cf78f-g848q\" (UID: \"0bde38ba-4188-4608-9c35-9fc65d033eb6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"
Sep 17 18:32:59.198293 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.198120    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bde38ba-4188-4608-9c35-9fc65d033eb6-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-bd5cf78f-g848q\" (UID: \"0bde38ba-4188-4608-9c35-9fc65d033eb6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"
Sep 17 18:32:59.198293 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.198191    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bde38ba-4188-4608-9c35-9fc65d033eb6-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-bd5cf78f-g848q\" (UID: \"0bde38ba-4188-4608-9c35-9fc65d033eb6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"
Sep 17 18:32:59.198293 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.198220    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bde38ba-4188-4608-9c35-9fc65d033eb6-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-bd5cf78f-g848q\" (UID: \"0bde38ba-4188-4608-9c35-9fc65d033eb6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"
Sep 17 18:32:59.198293 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.198266    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtmgc\" (UniqueName: \"kubernetes.io/projected/0bde38ba-4188-4608-9c35-9fc65d033eb6-kube-api-access-qtmgc\") pod \"lora-multiple-adapters-test-kserve-bd5cf78f-g848q\" (UID: \"0bde38ba-4188-4608-9c35-9fc65d033eb6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"
Sep 17 18:32:59.198293 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.198286    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bde38ba-4188-4608-9c35-9fc65d033eb6-model-cache\") pod \"lora-multiple-adapters-test-kserve-bd5cf78f-g848q\" (UID: \"0bde38ba-4188-4608-9c35-9fc65d033eb6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"
Sep 17 18:32:59.299288 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.299251    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bde38ba-4188-4608-9c35-9fc65d033eb6-model-cache\") pod \"lora-multiple-adapters-test-kserve-bd5cf78f-g848q\" (UID: \"0bde38ba-4188-4608-9c35-9fc65d033eb6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"
Sep 17 18:32:59.299494 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.299315    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bde38ba-4188-4608-9c35-9fc65d033eb6-home\") pod \"lora-multiple-adapters-test-kserve-bd5cf78f-g848q\" (UID: \"0bde38ba-4188-4608-9c35-9fc65d033eb6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"
Sep 17 18:32:59.299494 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.299423    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bde38ba-4188-4608-9c35-9fc65d033eb6-dshm\") pod \"lora-multiple-adapters-test-kserve-bd5cf78f-g848q\" (UID: \"0bde38ba-4188-4608-9c35-9fc65d033eb6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"
Sep 17 18:32:59.299494 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.299453    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bde38ba-4188-4608-9c35-9fc65d033eb6-tls-certs\") pod \"lora-multiple-adapters-test-kserve-bd5cf78f-g848q\" (UID: \"0bde38ba-4188-4608-9c35-9fc65d033eb6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"
Sep 17 18:32:59.299696 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.299495    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bde38ba-4188-4608-9c35-9fc65d033eb6-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-bd5cf78f-g848q\" (UID: \"0bde38ba-4188-4608-9c35-9fc65d033eb6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"
Sep 17 18:32:59.299696 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.299685    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bde38ba-4188-4608-9c35-9fc65d033eb6-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-bd5cf78f-g848q\" (UID: \"0bde38ba-4188-4608-9c35-9fc65d033eb6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"
Sep 17 18:32:59.299819 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.299723    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bde38ba-4188-4608-9c35-9fc65d033eb6-home\") pod \"lora-multiple-adapters-test-kserve-bd5cf78f-g848q\" (UID: \"0bde38ba-4188-4608-9c35-9fc65d033eb6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"
Sep 17 18:32:59.299819 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.299742    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bde38ba-4188-4608-9c35-9fc65d033eb6-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-bd5cf78f-g848q\" (UID: \"0bde38ba-4188-4608-9c35-9fc65d033eb6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"
Sep 17 18:32:59.299819 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.299778    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bde38ba-4188-4608-9c35-9fc65d033eb6-model-cache\") pod \"lora-multiple-adapters-test-kserve-bd5cf78f-g848q\" (UID: \"0bde38ba-4188-4608-9c35-9fc65d033eb6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"
Sep 17 18:32:59.299819 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.299807    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtmgc\" (UniqueName: \"kubernetes.io/projected/0bde38ba-4188-4608-9c35-9fc65d033eb6-kube-api-access-qtmgc\") pod \"lora-multiple-adapters-test-kserve-bd5cf78f-g848q\" (UID: \"0bde38ba-4188-4608-9c35-9fc65d033eb6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"
Sep 17 18:32:59.299819 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.299814    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bde38ba-4188-4608-9c35-9fc65d033eb6-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-bd5cf78f-g848q\" (UID: \"0bde38ba-4188-4608-9c35-9fc65d033eb6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"
Sep 17 18:32:59.300073 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.299990    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bde38ba-4188-4608-9c35-9fc65d033eb6-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-bd5cf78f-g848q\" (UID: \"0bde38ba-4188-4608-9c35-9fc65d033eb6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"
Sep 17 18:32:59.300493 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.300469    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bde38ba-4188-4608-9c35-9fc65d033eb6-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-bd5cf78f-g848q\" (UID: \"0bde38ba-4188-4608-9c35-9fc65d033eb6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"
Sep 17 18:32:59.302323 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.302301    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bde38ba-4188-4608-9c35-9fc65d033eb6-dshm\") pod \"lora-multiple-adapters-test-kserve-bd5cf78f-g848q\" (UID: \"0bde38ba-4188-4608-9c35-9fc65d033eb6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"
Sep 17 18:32:59.302570 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.302550    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bde38ba-4188-4608-9c35-9fc65d033eb6-tls-certs\") pod \"lora-multiple-adapters-test-kserve-bd5cf78f-g848q\" (UID: \"0bde38ba-4188-4608-9c35-9fc65d033eb6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"
Sep 17 18:32:59.309490 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.309465    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtmgc\" (UniqueName: \"kubernetes.io/projected/0bde38ba-4188-4608-9c35-9fc65d033eb6-kube-api-access-qtmgc\") pod \"lora-multiple-adapters-test-kserve-bd5cf78f-g848q\" (UID: \"0bde38ba-4188-4608-9c35-9fc65d033eb6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"
Sep 17 18:32:59.438246 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.438150    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"
Sep 17 18:32:59.723783 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:32:59.723752    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"]
Sep 17 18:32:59.724759 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:32:59.724730    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bde38ba_4188_4608_9c35_9fc65d033eb6.slice/crio-2c2edc4ff69426ee31cb92d386fb4de762b06f2ee0b2efc73d5c104c6bed5944 WatchSource:0}: Error finding container 2c2edc4ff69426ee31cb92d386fb4de762b06f2ee0b2efc73d5c104c6bed5944: Status 404 returned error can't find the container with id 2c2edc4ff69426ee31cb92d386fb4de762b06f2ee0b2efc73d5c104c6bed5944
Sep 17 18:33:00.509121 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:00.509081    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q" event={"ID":"0bde38ba-4188-4608-9c35-9fc65d033eb6","Type":"ContainerStarted","Data":"1e91aefd65c6daa3f9f0906c79c89b83de71fe67d2bf3c0864a5e58d8be4c8bb"}
Sep 17 18:33:00.509509 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:00.509153    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q" event={"ID":"0bde38ba-4188-4608-9c35-9fc65d033eb6","Type":"ContainerStarted","Data":"2c2edc4ff69426ee31cb92d386fb4de762b06f2ee0b2efc73d5c104c6bed5944"}
Sep 17 18:33:01.583546 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:01.583481    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2" podUID="fb46d7f3-4d8b-4716-b10f-d104098a0690" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8001/health\": dial tcp 10.134.0.54:8001: connect: connection refused"
Sep 17 18:33:03.524454 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:03.524418    2583 generic.go:356] "Generic (PLEG): container finished" podID="0bde38ba-4188-4608-9c35-9fc65d033eb6" containerID="1e91aefd65c6daa3f9f0906c79c89b83de71fe67d2bf3c0864a5e58d8be4c8bb" exitCode=0
Sep 17 18:33:03.524934 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:03.524490    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q" event={"ID":"0bde38ba-4188-4608-9c35-9fc65d033eb6","Type":"ContainerDied","Data":"1e91aefd65c6daa3f9f0906c79c89b83de71fe67d2bf3c0864a5e58d8be4c8bb"}
Sep 17 18:33:04.530385 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:04.530346    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q" event={"ID":"0bde38ba-4188-4608-9c35-9fc65d033eb6","Type":"ContainerStarted","Data":"9b25cb0abbc043bf1f3d92d8b159e43844752cd8631255460462696f9ccd2966"}
Sep 17 18:33:04.556383 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:04.556323    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q" podStartSLOduration=2.261967162 podStartE2EDuration="5.556309516s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.294342354s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:33:04.554072166 +0000 UTC m=+3495.645378941" watchObservedRunningTime="2026-09-17 18:33:04.556309516 +0000 UTC m=+3495.647616288"
Sep 17 18:33:05.438898 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:05.438866    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"
Sep 17 18:33:09.438481 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:09.438444    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"
Sep 17 18:33:09.440146 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:09.440119    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q" podUID="0bde38ba-4188-4608-9c35-9fc65d033eb6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 17 18:33:11.592067 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:11.592032    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:33:11.609936 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:11.609906    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:33:19.439604 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:19.439553    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q" podUID="0bde38ba-4188-4608-9c35-9fc65d033eb6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 17 18:33:22.009093 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:22.009059    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"]
Sep 17 18:33:29.439120 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:29.439075    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q" podUID="0bde38ba-4188-4608-9c35-9fc65d033eb6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 17 18:33:37.036755 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:37.036709    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2" podUID="fb46d7f3-4d8b-4716-b10f-d104098a0690" containerName="main" containerID="cri-o://1a0f1d30f17454db8ecdbd2b3c9aa805d352e90675c7f7b293ebe5248659d42d" gracePeriod=45
Sep 17 18:33:37.572244 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:37.572203    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2" podUID="fb46d7f3-4d8b-4716-b10f-d104098a0690" containerName="llm-d-routing-sidecar" containerID="cri-o://fe53438c270ee18a442afa2932f04611a9cb00b801071335e456e0c56ca5d270" gracePeriod=45
Sep 17 18:33:37.691791 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:37.691755    2583 generic.go:356] "Generic (PLEG): container finished" podID="fb46d7f3-4d8b-4716-b10f-d104098a0690" containerID="1a0f1d30f17454db8ecdbd2b3c9aa805d352e90675c7f7b293ebe5248659d42d" exitCode=0
Sep 17 18:33:37.691791 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:37.691785    2583 generic.go:356] "Generic (PLEG): container finished" podID="fb46d7f3-4d8b-4716-b10f-d104098a0690" containerID="fe53438c270ee18a442afa2932f04611a9cb00b801071335e456e0c56ca5d270" exitCode=0
Sep 17 18:33:37.691966 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:37.691841    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2" event={"ID":"fb46d7f3-4d8b-4716-b10f-d104098a0690","Type":"ContainerDied","Data":"1a0f1d30f17454db8ecdbd2b3c9aa805d352e90675c7f7b293ebe5248659d42d"}
Sep 17 18:33:37.691966 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:37.691924    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2" event={"ID":"fb46d7f3-4d8b-4716-b10f-d104098a0690","Type":"ContainerDied","Data":"fe53438c270ee18a442afa2932f04611a9cb00b801071335e456e0c56ca5d270"}
Sep 17 18:33:37.805590 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:37.805566    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:33:37.937641 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:37.937610    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb46d7f3-4d8b-4716-b10f-d104098a0690-model-cache\") pod \"fb46d7f3-4d8b-4716-b10f-d104098a0690\" (UID: \"fb46d7f3-4d8b-4716-b10f-d104098a0690\") "
Sep 17 18:33:37.937641 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:37.937643    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb46d7f3-4d8b-4716-b10f-d104098a0690-tls-certs\") pod \"fb46d7f3-4d8b-4716-b10f-d104098a0690\" (UID: \"fb46d7f3-4d8b-4716-b10f-d104098a0690\") "
Sep 17 18:33:37.937870 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:37.937688    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tmx2\" (UniqueName: \"kubernetes.io/projected/fb46d7f3-4d8b-4716-b10f-d104098a0690-kube-api-access-2tmx2\") pod \"fb46d7f3-4d8b-4716-b10f-d104098a0690\" (UID: \"fb46d7f3-4d8b-4716-b10f-d104098a0690\") "
Sep 17 18:33:37.937870 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:37.937711    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb46d7f3-4d8b-4716-b10f-d104098a0690-dshm\") pod \"fb46d7f3-4d8b-4716-b10f-d104098a0690\" (UID: \"fb46d7f3-4d8b-4716-b10f-d104098a0690\") "
Sep 17 18:33:37.937870 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:37.937734    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb46d7f3-4d8b-4716-b10f-d104098a0690-home\") pod \"fb46d7f3-4d8b-4716-b10f-d104098a0690\" (UID: \"fb46d7f3-4d8b-4716-b10f-d104098a0690\") "
Sep 17 18:33:37.937870 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:37.937787    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb46d7f3-4d8b-4716-b10f-d104098a0690-kserve-provision-location\") pod \"fb46d7f3-4d8b-4716-b10f-d104098a0690\" (UID: \"fb46d7f3-4d8b-4716-b10f-d104098a0690\") "
Sep 17 18:33:37.937870 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:37.937836    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb46d7f3-4d8b-4716-b10f-d104098a0690-cabundle-cert\") pod \"fb46d7f3-4d8b-4716-b10f-d104098a0690\" (UID: \"fb46d7f3-4d8b-4716-b10f-d104098a0690\") "
Sep 17 18:33:37.938143 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:37.937884    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb46d7f3-4d8b-4716-b10f-d104098a0690-tmp-dir\") pod \"fb46d7f3-4d8b-4716-b10f-d104098a0690\" (UID: \"fb46d7f3-4d8b-4716-b10f-d104098a0690\") "
Sep 17 18:33:37.938143 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:37.937909    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb46d7f3-4d8b-4716-b10f-d104098a0690-model-cache" (OuterVolumeSpecName: "model-cache") pod "fb46d7f3-4d8b-4716-b10f-d104098a0690" (UID: "fb46d7f3-4d8b-4716-b10f-d104098a0690"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:33:37.938143 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:37.938098    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb46d7f3-4d8b-4716-b10f-d104098a0690-model-cache\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:33:37.938449 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:37.938318    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb46d7f3-4d8b-4716-b10f-d104098a0690-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fb46d7f3-4d8b-4716-b10f-d104098a0690" (UID: "fb46d7f3-4d8b-4716-b10f-d104098a0690"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:33:37.938547 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:37.938495    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb46d7f3-4d8b-4716-b10f-d104098a0690-home" (OuterVolumeSpecName: "home") pod "fb46d7f3-4d8b-4716-b10f-d104098a0690" (UID: "fb46d7f3-4d8b-4716-b10f-d104098a0690"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:33:37.940342 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:37.940302    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb46d7f3-4d8b-4716-b10f-d104098a0690-dshm" (OuterVolumeSpecName: "dshm") pod "fb46d7f3-4d8b-4716-b10f-d104098a0690" (UID: "fb46d7f3-4d8b-4716-b10f-d104098a0690"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:33:37.940539 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:37.940511    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb46d7f3-4d8b-4716-b10f-d104098a0690-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fb46d7f3-4d8b-4716-b10f-d104098a0690" (UID: "fb46d7f3-4d8b-4716-b10f-d104098a0690"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:33:37.940645 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:37.940585    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb46d7f3-4d8b-4716-b10f-d104098a0690-kube-api-access-2tmx2" (OuterVolumeSpecName: "kube-api-access-2tmx2") pod "fb46d7f3-4d8b-4716-b10f-d104098a0690" (UID: "fb46d7f3-4d8b-4716-b10f-d104098a0690"). InnerVolumeSpecName "kube-api-access-2tmx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:33:37.952909 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:37.952883    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb46d7f3-4d8b-4716-b10f-d104098a0690-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fb46d7f3-4d8b-4716-b10f-d104098a0690" (UID: "fb46d7f3-4d8b-4716-b10f-d104098a0690"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:33:37.960589 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:37.960564    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb46d7f3-4d8b-4716-b10f-d104098a0690-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fb46d7f3-4d8b-4716-b10f-d104098a0690" (UID: "fb46d7f3-4d8b-4716-b10f-d104098a0690"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:33:38.039336 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:38.039277    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb46d7f3-4d8b-4716-b10f-d104098a0690-tmp-dir\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:33:38.039336 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:38.039300    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb46d7f3-4d8b-4716-b10f-d104098a0690-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:33:38.039336 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:38.039310    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tmx2\" (UniqueName: \"kubernetes.io/projected/fb46d7f3-4d8b-4716-b10f-d104098a0690-kube-api-access-2tmx2\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:33:38.039336 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:38.039319    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb46d7f3-4d8b-4716-b10f-d104098a0690-dshm\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:33:38.039336 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:38.039328    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb46d7f3-4d8b-4716-b10f-d104098a0690-home\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:33:38.039336 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:38.039335    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb46d7f3-4d8b-4716-b10f-d104098a0690-kserve-provision-location\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:33:38.039838 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:38.039345    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb46d7f3-4d8b-4716-b10f-d104098a0690-cabundle-cert\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:33:38.704991 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:38.704954    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2" event={"ID":"fb46d7f3-4d8b-4716-b10f-d104098a0690","Type":"ContainerDied","Data":"ccd105464eeed2bba0096121859f915dfad611370646df5505fdfeac1604aac5"}
Sep 17 18:33:38.704991 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:38.704982    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"
Sep 17 18:33:38.705222 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:38.705045    2583 scope.go:117] "RemoveContainer" containerID="1a0f1d30f17454db8ecdbd2b3c9aa805d352e90675c7f7b293ebe5248659d42d"
Sep 17 18:33:38.714310 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:38.714287    2583 scope.go:117] "RemoveContainer" containerID="9dacf5b7603b15c907a14f8f32a61b153360c557abd89d06ec917139aedd497f"
Sep 17 18:33:38.721597 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:38.721582    2583 scope.go:117] "RemoveContainer" containerID="fe53438c270ee18a442afa2932f04611a9cb00b801071335e456e0c56ca5d270"
Sep 17 18:33:38.729192 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:38.729161    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"]
Sep 17 18:33:38.735472 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:38.735447    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-765bc88f8c-bpmd2"]
Sep 17 18:33:39.042164 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:39.042084    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb46d7f3-4d8b-4716-b10f-d104098a0690" path="/var/lib/kubelet/pods/fb46d7f3-4d8b-4716-b10f-d104098a0690/volumes"
Sep 17 18:33:39.439541 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:39.439502    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q" podUID="0bde38ba-4188-4608-9c35-9fc65d033eb6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 17 18:33:49.438821 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:49.438776    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q" podUID="0bde38ba-4188-4608-9c35-9fc65d033eb6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 17 18:33:59.438909 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:33:59.438873    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q" podUID="0bde38ba-4188-4608-9c35-9fc65d033eb6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 17 18:34:09.439379 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:09.439331    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q" podUID="0bde38ba-4188-4608-9c35-9fc65d033eb6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 17 18:34:19.449074 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:19.449037    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"
Sep 17 18:34:19.456676 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:19.456639    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"
Sep 17 18:34:38.187577 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.187540    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"]
Sep 17 18:34:38.188268 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.188001    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb46d7f3-4d8b-4716-b10f-d104098a0690" containerName="main"
Sep 17 18:34:38.188268 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.188021    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb46d7f3-4d8b-4716-b10f-d104098a0690" containerName="main"
Sep 17 18:34:38.188268 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.188036    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb46d7f3-4d8b-4716-b10f-d104098a0690" containerName="llm-d-routing-sidecar"
Sep 17 18:34:38.188268 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.188044    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb46d7f3-4d8b-4716-b10f-d104098a0690" containerName="llm-d-routing-sidecar"
Sep 17 18:34:38.188268 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.188060    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb46d7f3-4d8b-4716-b10f-d104098a0690" containerName="storage-initializer"
Sep 17 18:34:38.188268 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.188068    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb46d7f3-4d8b-4716-b10f-d104098a0690" containerName="storage-initializer"
Sep 17 18:34:38.188268 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.188150    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb46d7f3-4d8b-4716-b10f-d104098a0690" containerName="main"
Sep 17 18:34:38.188268 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.188164    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb46d7f3-4d8b-4716-b10f-d104098a0690" containerName="llm-d-routing-sidecar"
Sep 17 18:34:38.191226 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.191205    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"
Sep 17 18:34:38.193757 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.193734    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 17 18:34:38.193982 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.193965    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 17 18:34:38.194073 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.193998    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 17 18:34:38.194227 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.194208    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:34:38.194335 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.194210    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 17 18:34:38.194335 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.194242    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-bwhqj\""
Sep 17 18:34:38.210584 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.210560    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"]
Sep 17 18:34:38.224157 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.224128    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87103603-7c26-42f0-b06d-6a980a2d7260-home\") pod \"router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r\" (UID: \"87103603-7c26-42f0-b06d-6a980a2d7260\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"
Sep 17 18:34:38.224281 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.224168    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87103603-7c26-42f0-b06d-6a980a2d7260-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r\" (UID: \"87103603-7c26-42f0-b06d-6a980a2d7260\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"
Sep 17 18:34:38.224281 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.224195    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87103603-7c26-42f0-b06d-6a980a2d7260-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r\" (UID: \"87103603-7c26-42f0-b06d-6a980a2d7260\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"
Sep 17 18:34:38.224281 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.224248    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87103603-7c26-42f0-b06d-6a980a2d7260-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r\" (UID: \"87103603-7c26-42f0-b06d-6a980a2d7260\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"
Sep 17 18:34:38.224419 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.224328    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87103603-7c26-42f0-b06d-6a980a2d7260-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r\" (UID: \"87103603-7c26-42f0-b06d-6a980a2d7260\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"
Sep 17 18:34:38.224419 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.224365    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm6r6\" (UniqueName: \"kubernetes.io/projected/87103603-7c26-42f0-b06d-6a980a2d7260-kube-api-access-gm6r6\") pod \"router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r\" (UID: \"87103603-7c26-42f0-b06d-6a980a2d7260\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"
Sep 17 18:34:38.224419 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.224397    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87103603-7c26-42f0-b06d-6a980a2d7260-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r\" (UID: \"87103603-7c26-42f0-b06d-6a980a2d7260\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"
Sep 17 18:34:38.224536 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.224429    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87103603-7c26-42f0-b06d-6a980a2d7260-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r\" (UID: \"87103603-7c26-42f0-b06d-6a980a2d7260\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"
Sep 17 18:34:38.325675 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.325619    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm6r6\" (UniqueName: \"kubernetes.io/projected/87103603-7c26-42f0-b06d-6a980a2d7260-kube-api-access-gm6r6\") pod \"router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r\" (UID: \"87103603-7c26-42f0-b06d-6a980a2d7260\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"
Sep 17 18:34:38.325855 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.325785    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87103603-7c26-42f0-b06d-6a980a2d7260-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r\" (UID: \"87103603-7c26-42f0-b06d-6a980a2d7260\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"
Sep 17 18:34:38.325855 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.325827    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87103603-7c26-42f0-b06d-6a980a2d7260-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r\" (UID: \"87103603-7c26-42f0-b06d-6a980a2d7260\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"
Sep 17 18:34:38.325936 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.325870    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87103603-7c26-42f0-b06d-6a980a2d7260-home\") pod \"router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r\" (UID: \"87103603-7c26-42f0-b06d-6a980a2d7260\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"
Sep 17 18:34:38.325936 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.325897    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87103603-7c26-42f0-b06d-6a980a2d7260-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r\" (UID: \"87103603-7c26-42f0-b06d-6a980a2d7260\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"
Sep 17 18:34:38.325936 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.325924    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87103603-7c26-42f0-b06d-6a980a2d7260-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r\" (UID: \"87103603-7c26-42f0-b06d-6a980a2d7260\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"
Sep 17 18:34:38.326096 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.325954    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87103603-7c26-42f0-b06d-6a980a2d7260-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r\" (UID: \"87103603-7c26-42f0-b06d-6a980a2d7260\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"
Sep 17 18:34:38.326096 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.326014    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87103603-7c26-42f0-b06d-6a980a2d7260-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r\" (UID: \"87103603-7c26-42f0-b06d-6a980a2d7260\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"
Sep 17 18:34:38.326327 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.326304    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87103603-7c26-42f0-b06d-6a980a2d7260-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r\" (UID: \"87103603-7c26-42f0-b06d-6a980a2d7260\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"
Sep 17 18:34:38.326434 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.326341    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87103603-7c26-42f0-b06d-6a980a2d7260-home\") pod \"router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r\" (UID: \"87103603-7c26-42f0-b06d-6a980a2d7260\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"
Sep 17 18:34:38.326434 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.326399    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87103603-7c26-42f0-b06d-6a980a2d7260-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r\" (UID: \"87103603-7c26-42f0-b06d-6a980a2d7260\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"
Sep 17 18:34:38.326631 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.326453    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87103603-7c26-42f0-b06d-6a980a2d7260-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r\" (UID: \"87103603-7c26-42f0-b06d-6a980a2d7260\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"
Sep 17 18:34:38.326770 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.326750    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87103603-7c26-42f0-b06d-6a980a2d7260-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r\" (UID: \"87103603-7c26-42f0-b06d-6a980a2d7260\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"
Sep 17 18:34:38.328056 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.328038    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87103603-7c26-42f0-b06d-6a980a2d7260-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r\" (UID: \"87103603-7c26-42f0-b06d-6a980a2d7260\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"
Sep 17 18:34:38.328455 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.328441    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87103603-7c26-42f0-b06d-6a980a2d7260-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r\" (UID: \"87103603-7c26-42f0-b06d-6a980a2d7260\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"
Sep 17 18:34:38.334951 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.334928    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm6r6\" (UniqueName: \"kubernetes.io/projected/87103603-7c26-42f0-b06d-6a980a2d7260-kube-api-access-gm6r6\") pod \"router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r\" (UID: \"87103603-7c26-42f0-b06d-6a980a2d7260\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"
Sep 17 18:34:38.501698 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.501596    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"
Sep 17 18:34:38.779698 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.779653    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"]
Sep 17 18:34:38.781594 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:34:38.781563    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87103603_7c26_42f0_b06d_6a980a2d7260.slice/crio-6cd33658bc38d254f8fc5e9bde3d5dae62000f5ecf7264218c60262128cb022d WatchSource:0}: Error finding container 6cd33658bc38d254f8fc5e9bde3d5dae62000f5ecf7264218c60262128cb022d: Status 404 returned error can't find the container with id 6cd33658bc38d254f8fc5e9bde3d5dae62000f5ecf7264218c60262128cb022d
Sep 17 18:34:38.974138 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.974095    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r" event={"ID":"87103603-7c26-42f0-b06d-6a980a2d7260","Type":"ContainerStarted","Data":"3fcde1239659f093405e120eb6f085c2e4743452ed66edfb5663aecdb6e98634"}
Sep 17 18:34:38.974306 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:38.974158    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r" event={"ID":"87103603-7c26-42f0-b06d-6a980a2d7260","Type":"ContainerStarted","Data":"6cd33658bc38d254f8fc5e9bde3d5dae62000f5ecf7264218c60262128cb022d"}
Sep 17 18:34:40.985997 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:40.985969    2583 generic.go:356] "Generic (PLEG): container finished" podID="87103603-7c26-42f0-b06d-6a980a2d7260" containerID="3fcde1239659f093405e120eb6f085c2e4743452ed66edfb5663aecdb6e98634" exitCode=0
Sep 17 18:34:40.986310 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:40.986038    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r" event={"ID":"87103603-7c26-42f0-b06d-6a980a2d7260","Type":"ContainerDied","Data":"3fcde1239659f093405e120eb6f085c2e4743452ed66edfb5663aecdb6e98634"}
Sep 17 18:34:41.992670 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:41.992626    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r" event={"ID":"87103603-7c26-42f0-b06d-6a980a2d7260","Type":"ContainerStarted","Data":"610d59b2c614d74e8586bf740ceba370d289000c508ea7e18a28667f5a80861e"}
Sep 17 18:34:42.020412 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:42.020359    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r" podStartSLOduration=1.890286169 podStartE2EDuration="4.020343512s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.130057343s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:34:42.017849767 +0000 UTC m=+3593.109156580" watchObservedRunningTime="2026-09-17 18:34:42.020343512 +0000 UTC m=+3593.111650276"
Sep 17 18:34:48.502745 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:48.502708    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"
Sep 17 18:34:48.503153 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:48.502778    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"
Sep 17 18:34:48.504454 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:48.504426    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r" podUID="87103603-7c26-42f0-b06d-6a980a2d7260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:34:58.502623 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:34:58.502575    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r" podUID="87103603-7c26-42f0-b06d-6a980a2d7260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:35:08.502821 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:08.502767    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r" podUID="87103603-7c26-42f0-b06d-6a980a2d7260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:35:18.502832 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:18.502787    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r" podUID="87103603-7c26-42f0-b06d-6a980a2d7260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:35:27.010611 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:27.010573    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-bd5cf78f-g848q_0bde38ba-4188-4608-9c35-9fc65d033eb6/storage-initializer/0.log"
Sep 17 18:35:27.020080 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:27.020045    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-bd5cf78f-g848q_0bde38ba-4188-4608-9c35-9fc65d033eb6/main/0.log"
Sep 17 18:35:28.050589 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:28.050547    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"]
Sep 17 18:35:28.502329 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:28.502275    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r" podUID="87103603-7c26-42f0-b06d-6a980a2d7260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:35:38.502089 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:38.502050    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r" podUID="87103603-7c26-42f0-b06d-6a980a2d7260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:35:43.072577 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:43.072513    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q" podUID="0bde38ba-4188-4608-9c35-9fc65d033eb6" containerName="main" containerID="cri-o://9b25cb0abbc043bf1f3d92d8b159e43844752cd8631255460462696f9ccd2966" gracePeriod=45
Sep 17 18:35:43.450588 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:43.450534    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q" podUID="0bde38ba-4188-4608-9c35-9fc65d033eb6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 17 18:35:43.728937 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:43.728908    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"
Sep 17 18:35:43.784739 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:43.784704    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bde38ba-4188-4608-9c35-9fc65d033eb6-tls-certs\") pod \"0bde38ba-4188-4608-9c35-9fc65d033eb6\" (UID: \"0bde38ba-4188-4608-9c35-9fc65d033eb6\") "
Sep 17 18:35:43.784883 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:43.784745    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtmgc\" (UniqueName: \"kubernetes.io/projected/0bde38ba-4188-4608-9c35-9fc65d033eb6-kube-api-access-qtmgc\") pod \"0bde38ba-4188-4608-9c35-9fc65d033eb6\" (UID: \"0bde38ba-4188-4608-9c35-9fc65d033eb6\") "
Sep 17 18:35:43.784883 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:43.784778    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bde38ba-4188-4608-9c35-9fc65d033eb6-tmp-dir\") pod \"0bde38ba-4188-4608-9c35-9fc65d033eb6\" (UID: \"0bde38ba-4188-4608-9c35-9fc65d033eb6\") "
Sep 17 18:35:43.784883 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:43.784795    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bde38ba-4188-4608-9c35-9fc65d033eb6-home\") pod \"0bde38ba-4188-4608-9c35-9fc65d033eb6\" (UID: \"0bde38ba-4188-4608-9c35-9fc65d033eb6\") "
Sep 17 18:35:43.784883 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:43.784810    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bde38ba-4188-4608-9c35-9fc65d033eb6-cabundle-cert\") pod \"0bde38ba-4188-4608-9c35-9fc65d033eb6\" (UID: \"0bde38ba-4188-4608-9c35-9fc65d033eb6\") "
Sep 17 18:35:43.784883 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:43.784841    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bde38ba-4188-4608-9c35-9fc65d033eb6-kserve-provision-location\") pod \"0bde38ba-4188-4608-9c35-9fc65d033eb6\" (UID: \"0bde38ba-4188-4608-9c35-9fc65d033eb6\") "
Sep 17 18:35:43.784883 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:43.784871    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bde38ba-4188-4608-9c35-9fc65d033eb6-model-cache\") pod \"0bde38ba-4188-4608-9c35-9fc65d033eb6\" (UID: \"0bde38ba-4188-4608-9c35-9fc65d033eb6\") "
Sep 17 18:35:43.785152 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:43.784898    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bde38ba-4188-4608-9c35-9fc65d033eb6-dshm\") pod \"0bde38ba-4188-4608-9c35-9fc65d033eb6\" (UID: \"0bde38ba-4188-4608-9c35-9fc65d033eb6\") "
Sep 17 18:35:43.785216 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:43.785151    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bde38ba-4188-4608-9c35-9fc65d033eb6-home" (OuterVolumeSpecName: "home") pod "0bde38ba-4188-4608-9c35-9fc65d033eb6" (UID: "0bde38ba-4188-4608-9c35-9fc65d033eb6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:35:43.785599 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:43.785329    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bde38ba-4188-4608-9c35-9fc65d033eb6-model-cache" (OuterVolumeSpecName: "model-cache") pod "0bde38ba-4188-4608-9c35-9fc65d033eb6" (UID: "0bde38ba-4188-4608-9c35-9fc65d033eb6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:35:43.785848 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:43.785813    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bde38ba-4188-4608-9c35-9fc65d033eb6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0bde38ba-4188-4608-9c35-9fc65d033eb6" (UID: "0bde38ba-4188-4608-9c35-9fc65d033eb6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:35:43.787533 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:43.787502    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bde38ba-4188-4608-9c35-9fc65d033eb6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0bde38ba-4188-4608-9c35-9fc65d033eb6" (UID: "0bde38ba-4188-4608-9c35-9fc65d033eb6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:35:43.787709 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:43.787679    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bde38ba-4188-4608-9c35-9fc65d033eb6-kube-api-access-qtmgc" (OuterVolumeSpecName: "kube-api-access-qtmgc") pod "0bde38ba-4188-4608-9c35-9fc65d033eb6" (UID: "0bde38ba-4188-4608-9c35-9fc65d033eb6"). InnerVolumeSpecName "kube-api-access-qtmgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:35:43.787709 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:43.787688    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bde38ba-4188-4608-9c35-9fc65d033eb6-dshm" (OuterVolumeSpecName: "dshm") pod "0bde38ba-4188-4608-9c35-9fc65d033eb6" (UID: "0bde38ba-4188-4608-9c35-9fc65d033eb6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:35:43.797020 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:43.796993    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bde38ba-4188-4608-9c35-9fc65d033eb6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0bde38ba-4188-4608-9c35-9fc65d033eb6" (UID: "0bde38ba-4188-4608-9c35-9fc65d033eb6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:35:43.809941 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:43.809913    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bde38ba-4188-4608-9c35-9fc65d033eb6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0bde38ba-4188-4608-9c35-9fc65d033eb6" (UID: "0bde38ba-4188-4608-9c35-9fc65d033eb6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:35:43.886507 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:43.886478    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bde38ba-4188-4608-9c35-9fc65d033eb6-tmp-dir\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:35:43.886507 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:43.886501    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bde38ba-4188-4608-9c35-9fc65d033eb6-home\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:35:43.886507 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:43.886510    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bde38ba-4188-4608-9c35-9fc65d033eb6-cabundle-cert\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:35:43.886717 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:43.886520    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bde38ba-4188-4608-9c35-9fc65d033eb6-kserve-provision-location\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:35:43.886717 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:43.886530    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bde38ba-4188-4608-9c35-9fc65d033eb6-model-cache\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:35:43.886717 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:43.886538    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bde38ba-4188-4608-9c35-9fc65d033eb6-dshm\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:35:43.886717 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:43.886545    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bde38ba-4188-4608-9c35-9fc65d033eb6-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:35:43.886717 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:43.886552    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtmgc\" (UniqueName: \"kubernetes.io/projected/0bde38ba-4188-4608-9c35-9fc65d033eb6-kube-api-access-qtmgc\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:35:44.288316 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:44.288279    2583 generic.go:356] "Generic (PLEG): container finished" podID="0bde38ba-4188-4608-9c35-9fc65d033eb6" containerID="9b25cb0abbc043bf1f3d92d8b159e43844752cd8631255460462696f9ccd2966" exitCode=0
Sep 17 18:35:44.288778 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:44.288328    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"
Sep 17 18:35:44.288778 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:44.288356    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q" event={"ID":"0bde38ba-4188-4608-9c35-9fc65d033eb6","Type":"ContainerDied","Data":"9b25cb0abbc043bf1f3d92d8b159e43844752cd8631255460462696f9ccd2966"}
Sep 17 18:35:44.288778 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:44.288410    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q" event={"ID":"0bde38ba-4188-4608-9c35-9fc65d033eb6","Type":"ContainerDied","Data":"2c2edc4ff69426ee31cb92d386fb4de762b06f2ee0b2efc73d5c104c6bed5944"}
Sep 17 18:35:44.288778 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:44.288433    2583 scope.go:117] "RemoveContainer" containerID="9b25cb0abbc043bf1f3d92d8b159e43844752cd8631255460462696f9ccd2966"
Sep 17 18:35:44.297400 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:44.297383    2583 scope.go:117] "RemoveContainer" containerID="1e91aefd65c6daa3f9f0906c79c89b83de71fe67d2bf3c0864a5e58d8be4c8bb"
Sep 17 18:35:44.307449 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:44.307432    2583 scope.go:117] "RemoveContainer" containerID="9b25cb0abbc043bf1f3d92d8b159e43844752cd8631255460462696f9ccd2966"
Sep 17 18:35:44.307731 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:35:44.307714    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b25cb0abbc043bf1f3d92d8b159e43844752cd8631255460462696f9ccd2966\": container with ID starting with 9b25cb0abbc043bf1f3d92d8b159e43844752cd8631255460462696f9ccd2966 not found: ID does not exist" containerID="9b25cb0abbc043bf1f3d92d8b159e43844752cd8631255460462696f9ccd2966"
Sep 17 18:35:44.307808 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:44.307738    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b25cb0abbc043bf1f3d92d8b159e43844752cd8631255460462696f9ccd2966"} err="failed to get container status \"9b25cb0abbc043bf1f3d92d8b159e43844752cd8631255460462696f9ccd2966\": rpc error: code = NotFound desc = could not find container \"9b25cb0abbc043bf1f3d92d8b159e43844752cd8631255460462696f9ccd2966\": container with ID starting with 9b25cb0abbc043bf1f3d92d8b159e43844752cd8631255460462696f9ccd2966 not found: ID does not exist"
Sep 17 18:35:44.307808 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:44.307754    2583 scope.go:117] "RemoveContainer" containerID="1e91aefd65c6daa3f9f0906c79c89b83de71fe67d2bf3c0864a5e58d8be4c8bb"
Sep 17 18:35:44.308015 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:35:44.307998    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e91aefd65c6daa3f9f0906c79c89b83de71fe67d2bf3c0864a5e58d8be4c8bb\": container with ID starting with 1e91aefd65c6daa3f9f0906c79c89b83de71fe67d2bf3c0864a5e58d8be4c8bb not found: ID does not exist" containerID="1e91aefd65c6daa3f9f0906c79c89b83de71fe67d2bf3c0864a5e58d8be4c8bb"
Sep 17 18:35:44.308055 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:44.308020    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e91aefd65c6daa3f9f0906c79c89b83de71fe67d2bf3c0864a5e58d8be4c8bb"} err="failed to get container status \"1e91aefd65c6daa3f9f0906c79c89b83de71fe67d2bf3c0864a5e58d8be4c8bb\": rpc error: code = NotFound desc = could not find container \"1e91aefd65c6daa3f9f0906c79c89b83de71fe67d2bf3c0864a5e58d8be4c8bb\": container with ID starting with 1e91aefd65c6daa3f9f0906c79c89b83de71fe67d2bf3c0864a5e58d8be4c8bb not found: ID does not exist"
Sep 17 18:35:44.312275 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:44.312252    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"]
Sep 17 18:35:44.316264 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:44.316245    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-bd5cf78f-g848q"]
Sep 17 18:35:45.040836 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:45.040804    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bde38ba-4188-4608-9c35-9fc65d033eb6" path="/var/lib/kubelet/pods/0bde38ba-4188-4608-9c35-9fc65d033eb6/volumes"
Sep 17 18:35:48.503033 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:48.502990    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r" podUID="87103603-7c26-42f0-b06d-6a980a2d7260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:35:51.623529 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:51.623493    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-tvmm2"]
Sep 17 18:35:51.623961 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:51.623817    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bde38ba-4188-4608-9c35-9fc65d033eb6" containerName="main"
Sep 17 18:35:51.623961 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:51.623830    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bde38ba-4188-4608-9c35-9fc65d033eb6" containerName="main"
Sep 17 18:35:51.623961 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:51.623840    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bde38ba-4188-4608-9c35-9fc65d033eb6" containerName="storage-initializer"
Sep 17 18:35:51.623961 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:51.623845    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bde38ba-4188-4608-9c35-9fc65d033eb6" containerName="storage-initializer"
Sep 17 18:35:51.623961 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:51.623897    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bde38ba-4188-4608-9c35-9fc65d033eb6" containerName="main"
Sep 17 18:35:51.627180 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:51.627163    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-tvmm2"
Sep 17 18:35:51.629845 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:51.629823    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 17 18:35:51.630825 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:51.630807    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 17 18:35:51.630919 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:51.630831    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 17 18:35:51.630919 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:51.630837    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-crz5l\""
Sep 17 18:35:51.634299 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:51.634197    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-tvmm2"]
Sep 17 18:35:51.759696 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:51.759609    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pfhg\" (UniqueName: \"kubernetes.io/projected/aa5f1843-76b6-42c5-8afb-2e00fefeff4e-kube-api-access-2pfhg\") pod \"e2e-pvc-lora-download-tvmm2\" (UID: \"aa5f1843-76b6-42c5-8afb-2e00fefeff4e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-tvmm2"
Sep 17 18:35:51.860596 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:51.860568    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pfhg\" (UniqueName: \"kubernetes.io/projected/aa5f1843-76b6-42c5-8afb-2e00fefeff4e-kube-api-access-2pfhg\") pod \"e2e-pvc-lora-download-tvmm2\" (UID: \"aa5f1843-76b6-42c5-8afb-2e00fefeff4e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-tvmm2"
Sep 17 18:35:51.869093 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:51.869067    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pfhg\" (UniqueName: \"kubernetes.io/projected/aa5f1843-76b6-42c5-8afb-2e00fefeff4e-kube-api-access-2pfhg\") pod \"e2e-pvc-lora-download-tvmm2\" (UID: \"aa5f1843-76b6-42c5-8afb-2e00fefeff4e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-tvmm2"
Sep 17 18:35:56.098802 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:56.098755    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-048a3c21-2fe1-48d8-9c35-8c255bd034d8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05043ab4858da5f0e\") pod \"e2e-pvc-lora-download-tvmm2\" (UID: \"aa5f1843-76b6-42c5-8afb-2e00fefeff4e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-tvmm2"
Sep 17 18:35:56.104530 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:56.104508    2583 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-048a3c21-2fe1-48d8-9c35-8c255bd034d8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05043ab4858da5f0e\") pod \"e2e-pvc-lora-download-tvmm2\" (UID: \"aa5f1843-76b6-42c5-8afb-2e00fefeff4e\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-tvmm2"
Sep 17 18:35:56.199640 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:56.199613    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-048a3c21-2fe1-48d8-9c35-8c255bd034d8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05043ab4858da5f0e\") pod \"e2e-pvc-lora-download-tvmm2\" (UID: \"aa5f1843-76b6-42c5-8afb-2e00fefeff4e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-tvmm2"
Sep 17 18:35:56.199792 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:56.199763    2583 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-048a3c21-2fe1-48d8-9c35-8c255bd034d8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05043ab4858da5f0e\") pod \"e2e-pvc-lora-download-tvmm2\" (UID: \"aa5f1843-76b6-42c5-8afb-2e00fefeff4e\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-tvmm2"
Sep 17 18:35:56.203337 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:56.203314    2583 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-048a3c21-2fe1-48d8-9c35-8c255bd034d8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05043ab4858da5f0e\") pod \"e2e-pvc-lora-download-tvmm2\" (UID: \"aa5f1843-76b6-42c5-8afb-2e00fefeff4e\") DevicePath \"csi-964e7867a9e2c3c7a0c2e42abe77980f25c3fb763de53291fa381acca5db41f5\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-tvmm2"
Sep 17 18:35:56.419768 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:56.419730    2583 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-048a3c21-2fe1-48d8-9c35-8c255bd034d8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05043ab4858da5f0e\") pod \"e2e-pvc-lora-download-tvmm2\" (UID: \"aa5f1843-76b6-42c5-8afb-2e00fefeff4e\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/929e916b279a872027069494247fd75baa23ed2eae2d984e10f83e67c2eb05e8/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-tvmm2"
Sep 17 18:35:56.432913 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:56.432888    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-048a3c21-2fe1-48d8-9c35-8c255bd034d8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05043ab4858da5f0e\") pod \"e2e-pvc-lora-download-tvmm2\" (UID: \"aa5f1843-76b6-42c5-8afb-2e00fefeff4e\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-tvmm2"
Sep 17 18:35:56.437336 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:56.437315    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-tvmm2"
Sep 17 18:35:56.707470 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:56.707375    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-tvmm2"]
Sep 17 18:35:56.710490 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:35:56.710462    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa5f1843_76b6_42c5_8afb_2e00fefeff4e.slice/crio-33c0fa1e2ee8c9ce6c9531bda07eca7dcc3b533075ffebdaaf66a487c21950a8 WatchSource:0}: Error finding container 33c0fa1e2ee8c9ce6c9531bda07eca7dcc3b533075ffebdaaf66a487c21950a8: Status 404 returned error can't find the container with id 33c0fa1e2ee8c9ce6c9531bda07eca7dcc3b533075ffebdaaf66a487c21950a8
Sep 17 18:35:57.348996 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:57.348961    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-tvmm2" event={"ID":"aa5f1843-76b6-42c5-8afb-2e00fefeff4e","Type":"ContainerStarted","Data":"2f03621d6be0d5c0100a5f5357abf3481aa924ab757b27b10b31371921963430"}
Sep 17 18:35:57.348996 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:57.349001    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-tvmm2" event={"ID":"aa5f1843-76b6-42c5-8afb-2e00fefeff4e","Type":"ContainerStarted","Data":"33c0fa1e2ee8c9ce6c9531bda07eca7dcc3b533075ffebdaaf66a487c21950a8"}
Sep 17 18:35:57.365430 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:57.365383    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-tvmm2" podStartSLOduration=9.365368417 podStartE2EDuration="9.365368417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:35:57.363525208 +0000 UTC m=+3668.454831972" watchObservedRunningTime="2026-09-17 18:35:57.365368417 +0000 UTC m=+3668.456675182"
Sep 17 18:35:58.502678 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:58.502618    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r" podUID="87103603-7c26-42f0-b06d-6a980a2d7260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:35:59.360259 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:59.360225    2583 generic.go:356] "Generic (PLEG): container finished" podID="aa5f1843-76b6-42c5-8afb-2e00fefeff4e" containerID="2f03621d6be0d5c0100a5f5357abf3481aa924ab757b27b10b31371921963430" exitCode=0
Sep 17 18:35:59.360464 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:35:59.360297    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-tvmm2" event={"ID":"aa5f1843-76b6-42c5-8afb-2e00fefeff4e","Type":"ContainerDied","Data":"2f03621d6be0d5c0100a5f5357abf3481aa924ab757b27b10b31371921963430"}
Sep 17 18:36:00.494325 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:00.494290    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-tvmm2"
Sep 17 18:36:00.639279 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:00.639185    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pfhg\" (UniqueName: \"kubernetes.io/projected/aa5f1843-76b6-42c5-8afb-2e00fefeff4e-kube-api-access-2pfhg\") pod \"aa5f1843-76b6-42c5-8afb-2e00fefeff4e\" (UID: \"aa5f1843-76b6-42c5-8afb-2e00fefeff4e\") "
Sep 17 18:36:00.639472 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:00.639313    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05043ab4858da5f0e\") pod \"aa5f1843-76b6-42c5-8afb-2e00fefeff4e\" (UID: \"aa5f1843-76b6-42c5-8afb-2e00fefeff4e\") "
Sep 17 18:36:00.641610 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:00.641584    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa5f1843-76b6-42c5-8afb-2e00fefeff4e-kube-api-access-2pfhg" (OuterVolumeSpecName: "kube-api-access-2pfhg") pod "aa5f1843-76b6-42c5-8afb-2e00fefeff4e" (UID: "aa5f1843-76b6-42c5-8afb-2e00fefeff4e"). InnerVolumeSpecName "kube-api-access-2pfhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:36:00.643516 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:00.643487    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-05043ab4858da5f0e" (OuterVolumeSpecName: "model-storage") pod "aa5f1843-76b6-42c5-8afb-2e00fefeff4e" (UID: "aa5f1843-76b6-42c5-8afb-2e00fefeff4e"). InnerVolumeSpecName "pvc-048a3c21-2fe1-48d8-9c35-8c255bd034d8". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 18:36:00.740220 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:00.740185    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pfhg\" (UniqueName: \"kubernetes.io/projected/aa5f1843-76b6-42c5-8afb-2e00fefeff4e-kube-api-access-2pfhg\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:36:00.740406 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:00.740241    2583 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-048a3c21-2fe1-48d8-9c35-8c255bd034d8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05043ab4858da5f0e\") on node \"ip-10-0-141-30.ec2.internal\" "
Sep 17 18:36:01.371532 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:01.371499    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-tvmm2" event={"ID":"aa5f1843-76b6-42c5-8afb-2e00fefeff4e","Type":"ContainerDied","Data":"33c0fa1e2ee8c9ce6c9531bda07eca7dcc3b533075ffebdaaf66a487c21950a8"}
Sep 17 18:36:01.371715 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:01.371539    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33c0fa1e2ee8c9ce6c9531bda07eca7dcc3b533075ffebdaaf66a487c21950a8"
Sep 17 18:36:01.371715 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:01.371546    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-tvmm2"
Sep 17 18:36:01.710670 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:01.710636    2583 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-048a3c21-2fe1-48d8-9c35-8c255bd034d8" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-05043ab4858da5f0e") on node "ip-10-0-141-30.ec2.internal"
Sep 17 18:36:01.748167 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:01.748124    2583 reconciler_common.go:299] "Volume detached for volume \"pvc-048a3c21-2fe1-48d8-9c35-8c255bd034d8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05043ab4858da5f0e\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"csi-964e7867a9e2c3c7a0c2e42abe77980f25c3fb763de53291fa381acca5db41f5\""
Sep 17 18:36:08.503046 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:08.503004    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r" podUID="87103603-7c26-42f0-b06d-6a980a2d7260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:36:18.502294 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:18.502247    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r" podUID="87103603-7c26-42f0-b06d-6a980a2d7260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:36:21.418537 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.418497    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"]
Sep 17 18:36:21.418983 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.418964    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa5f1843-76b6-42c5-8afb-2e00fefeff4e" containerName="storage-initializer"
Sep 17 18:36:21.419028 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.418987    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa5f1843-76b6-42c5-8afb-2e00fefeff4e" containerName="storage-initializer"
Sep 17 18:36:21.419080 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.419069    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa5f1843-76b6-42c5-8afb-2e00fefeff4e" containerName="storage-initializer"
Sep 17 18:36:21.424964 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.424946    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:21.427517 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.427488    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:36:21.427654 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.427526    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 17 18:36:21.427759 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.427695    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 17 18:36:21.427825 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.427779    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-crz5l\""
Sep 17 18:36:21.428019 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.428003    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 17 18:36:21.428338 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.428324    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"lora-pvc-single-adapter-test-kserve-self-signed-certs\""
Sep 17 18:36:21.436854 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.436824    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"]
Sep 17 18:36:21.513230 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.513196    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1c90212-326c-452f-9831-e8a4a13936e9-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-7c7446d756-568f2\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:21.513230 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.513230    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1c90212-326c-452f-9831-e8a4a13936e9-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-7c7446d756-568f2\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:21.513462 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.513252    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1c90212-326c-452f-9831-e8a4a13936e9-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-7c7446d756-568f2\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:21.513462 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.513298    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1c90212-326c-452f-9831-e8a4a13936e9-home\") pod \"lora-pvc-single-adapter-test-kserve-7c7446d756-568f2\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:21.513462 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.513335    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1c90212-326c-452f-9831-e8a4a13936e9-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-7c7446d756-568f2\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:21.513462 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.513362    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1c90212-326c-452f-9831-e8a4a13936e9-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-7c7446d756-568f2\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:21.513462 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.513387    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1c90212-326c-452f-9831-e8a4a13936e9-dshm\") pod \"lora-pvc-single-adapter-test-kserve-7c7446d756-568f2\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:21.614436 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.614394    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1c90212-326c-452f-9831-e8a4a13936e9-dshm\") pod \"lora-pvc-single-adapter-test-kserve-7c7446d756-568f2\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:21.614630 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.614478    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1c90212-326c-452f-9831-e8a4a13936e9-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-7c7446d756-568f2\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:21.614630 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.614507    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1c90212-326c-452f-9831-e8a4a13936e9-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-7c7446d756-568f2\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:21.614630 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.614531    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1c90212-326c-452f-9831-e8a4a13936e9-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-7c7446d756-568f2\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:21.614630 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.614560    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1c90212-326c-452f-9831-e8a4a13936e9-home\") pod \"lora-pvc-single-adapter-test-kserve-7c7446d756-568f2\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:21.614880 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.614698    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6pbg\" (UniqueName: \"kubernetes.io/projected/f1c90212-326c-452f-9831-e8a4a13936e9-kube-api-access-x6pbg\") pod \"lora-pvc-single-adapter-test-kserve-7c7446d756-568f2\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:21.614880 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.614750    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1c90212-326c-452f-9831-e8a4a13936e9-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-7c7446d756-568f2\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:21.614880 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.614788    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1c90212-326c-452f-9831-e8a4a13936e9-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-7c7446d756-568f2\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:21.615041 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.614958    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1c90212-326c-452f-9831-e8a4a13936e9-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-7c7446d756-568f2\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:21.615041 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.615022    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1c90212-326c-452f-9831-e8a4a13936e9-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-7c7446d756-568f2\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:21.615221 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.615195    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1c90212-326c-452f-9831-e8a4a13936e9-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-7c7446d756-568f2\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:21.615416 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.615246    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1c90212-326c-452f-9831-e8a4a13936e9-home\") pod \"lora-pvc-single-adapter-test-kserve-7c7446d756-568f2\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:21.615632 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.615606    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1c90212-326c-452f-9831-e8a4a13936e9-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-7c7446d756-568f2\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:21.616841 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.616822    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1c90212-326c-452f-9831-e8a4a13936e9-dshm\") pod \"lora-pvc-single-adapter-test-kserve-7c7446d756-568f2\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:21.617119 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.617101    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1c90212-326c-452f-9831-e8a4a13936e9-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-7c7446d756-568f2\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:21.715728 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.715632    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6pbg\" (UniqueName: \"kubernetes.io/projected/f1c90212-326c-452f-9831-e8a4a13936e9-kube-api-access-x6pbg\") pod \"lora-pvc-single-adapter-test-kserve-7c7446d756-568f2\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:21.724892 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:21.724869    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6pbg\" (UniqueName: \"kubernetes.io/projected/f1c90212-326c-452f-9831-e8a4a13936e9-kube-api-access-x6pbg\") pod \"lora-pvc-single-adapter-test-kserve-7c7446d756-568f2\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:24.035335 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:24.035289    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-048a3c21-2fe1-48d8-9c35-8c255bd034d8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05043ab4858da5f0e\") pod \"lora-pvc-single-adapter-test-kserve-7c7446d756-568f2\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:24.040109 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:24.040084    2583 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-048a3c21-2fe1-48d8-9c35-8c255bd034d8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05043ab4858da5f0e\") pod \"lora-pvc-single-adapter-test-kserve-7c7446d756-568f2\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:24.136326 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:24.136276    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-048a3c21-2fe1-48d8-9c35-8c255bd034d8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05043ab4858da5f0e\") pod \"lora-pvc-single-adapter-test-kserve-7c7446d756-568f2\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:24.136539 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:24.136384    2583 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-048a3c21-2fe1-48d8-9c35-8c255bd034d8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05043ab4858da5f0e\") pod \"lora-pvc-single-adapter-test-kserve-7c7446d756-568f2\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:24.140416 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:24.140387    2583 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-048a3c21-2fe1-48d8-9c35-8c255bd034d8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05043ab4858da5f0e\") pod \"lora-pvc-single-adapter-test-kserve-7c7446d756-568f2\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") DevicePath \"csi-964e7867a9e2c3c7a0c2e42abe77980f25c3fb763de53291fa381acca5db41f5\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:24.234248 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:24.234208    2583 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-048a3c21-2fe1-48d8-9c35-8c255bd034d8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05043ab4858da5f0e\") pod \"lora-pvc-single-adapter-test-kserve-7c7446d756-568f2\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/929e916b279a872027069494247fd75baa23ed2eae2d984e10f83e67c2eb05e8/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:24.252989 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:24.252959    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-048a3c21-2fe1-48d8-9c35-8c255bd034d8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05043ab4858da5f0e\") pod \"lora-pvc-single-adapter-test-kserve-7c7446d756-568f2\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:24.438225 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:24.438190    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:24.718455 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:24.718376    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"]
Sep 17 18:36:24.721089 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:36:24.721059    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c90212_326c_452f_9831_e8a4a13936e9.slice/crio-cd7b66e31c2879b1108ece0ed90e4f0675fd11d2affedb58a4e607e0eff39235 WatchSource:0}: Error finding container cd7b66e31c2879b1108ece0ed90e4f0675fd11d2affedb58a4e607e0eff39235: Status 404 returned error can't find the container with id cd7b66e31c2879b1108ece0ed90e4f0675fd11d2affedb58a4e607e0eff39235
Sep 17 18:36:25.483911 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:25.483876    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2" event={"ID":"f1c90212-326c-452f-9831-e8a4a13936e9","Type":"ContainerStarted","Data":"0b78d4abdaaaab1b558ff43eb6dc46a2e3cb6920ed0bed91cf37193313d8d28a"}
Sep 17 18:36:25.484446 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:25.483942    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2" event={"ID":"f1c90212-326c-452f-9831-e8a4a13936e9","Type":"ContainerStarted","Data":"cd7b66e31c2879b1108ece0ed90e4f0675fd11d2affedb58a4e607e0eff39235"}
Sep 17 18:36:27.495839 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:27.495806    2583 generic.go:356] "Generic (PLEG): container finished" podID="f1c90212-326c-452f-9831-e8a4a13936e9" containerID="0b78d4abdaaaab1b558ff43eb6dc46a2e3cb6920ed0bed91cf37193313d8d28a" exitCode=0
Sep 17 18:36:27.496256 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:27.495875    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2" event={"ID":"f1c90212-326c-452f-9831-e8a4a13936e9","Type":"ContainerDied","Data":"0b78d4abdaaaab1b558ff43eb6dc46a2e3cb6920ed0bed91cf37193313d8d28a"}
Sep 17 18:36:28.502636 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:28.502592    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r" podUID="87103603-7c26-42f0-b06d-6a980a2d7260" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Sep 17 18:36:28.505484 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:28.505450    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2" event={"ID":"f1c90212-326c-452f-9831-e8a4a13936e9","Type":"ContainerStarted","Data":"adb2b07a80df337b056461a1b4e63cf474d3afe96f680a001a6148dca7f99146"}
Sep 17 18:36:28.534632 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:28.534587    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2" podStartSLOduration=5.288516641 podStartE2EDuration="7.534572569s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.246055928s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:36:28.533027225 +0000 UTC m=+3699.624334002" watchObservedRunningTime="2026-09-17 18:36:28.534572569 +0000 UTC m=+3699.625879333"
Sep 17 18:36:29.439140 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:29.439103    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:34.438914 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:34.438882    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:36:34.440444 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:34.440414    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2" podUID="f1c90212-326c-452f-9831-e8a4a13936e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 17 18:36:38.512647 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:38.512606    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"
Sep 17 18:36:38.520957 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:38.520927    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"
Sep 17 18:36:44.438972 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:44.438926    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2" podUID="f1c90212-326c-452f-9831-e8a4a13936e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 17 18:36:54.439208 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:54.439155    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2" podUID="f1c90212-326c-452f-9831-e8a4a13936e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 17 18:36:57.234016 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:36:57.233984    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"]
Sep 17 18:37:04.439141 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:04.439088    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2" podUID="f1c90212-326c-452f-9831-e8a4a13936e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 17 18:37:12.256362 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:12.256318    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r" podUID="87103603-7c26-42f0-b06d-6a980a2d7260" containerName="main" containerID="cri-o://610d59b2c614d74e8586bf740ceba370d289000c508ea7e18a28667f5a80861e" gracePeriod=45
Sep 17 18:37:13.002997 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.002972    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"
Sep 17 18:37:13.166369 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.166338    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87103603-7c26-42f0-b06d-6a980a2d7260-model-cache\") pod \"87103603-7c26-42f0-b06d-6a980a2d7260\" (UID: \"87103603-7c26-42f0-b06d-6a980a2d7260\") "
Sep 17 18:37:13.166684 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.166388    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87103603-7c26-42f0-b06d-6a980a2d7260-tmp-dir\") pod \"87103603-7c26-42f0-b06d-6a980a2d7260\" (UID: \"87103603-7c26-42f0-b06d-6a980a2d7260\") "
Sep 17 18:37:13.166684 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.166434    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87103603-7c26-42f0-b06d-6a980a2d7260-dshm\") pod \"87103603-7c26-42f0-b06d-6a980a2d7260\" (UID: \"87103603-7c26-42f0-b06d-6a980a2d7260\") "
Sep 17 18:37:13.166684 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.166460    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87103603-7c26-42f0-b06d-6a980a2d7260-kserve-provision-location\") pod \"87103603-7c26-42f0-b06d-6a980a2d7260\" (UID: \"87103603-7c26-42f0-b06d-6a980a2d7260\") "
Sep 17 18:37:13.166684 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.166491    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87103603-7c26-42f0-b06d-6a980a2d7260-home\") pod \"87103603-7c26-42f0-b06d-6a980a2d7260\" (UID: \"87103603-7c26-42f0-b06d-6a980a2d7260\") "
Sep 17 18:37:13.166684 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.166516    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87103603-7c26-42f0-b06d-6a980a2d7260-tls-certs\") pod \"87103603-7c26-42f0-b06d-6a980a2d7260\" (UID: \"87103603-7c26-42f0-b06d-6a980a2d7260\") "
Sep 17 18:37:13.166684 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.166588    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gm6r6\" (UniqueName: \"kubernetes.io/projected/87103603-7c26-42f0-b06d-6a980a2d7260-kube-api-access-gm6r6\") pod \"87103603-7c26-42f0-b06d-6a980a2d7260\" (UID: \"87103603-7c26-42f0-b06d-6a980a2d7260\") "
Sep 17 18:37:13.166684 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.166655    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87103603-7c26-42f0-b06d-6a980a2d7260-cabundle-cert\") pod \"87103603-7c26-42f0-b06d-6a980a2d7260\" (UID: \"87103603-7c26-42f0-b06d-6a980a2d7260\") "
Sep 17 18:37:13.167173 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.167143    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87103603-7c26-42f0-b06d-6a980a2d7260-model-cache" (OuterVolumeSpecName: "model-cache") pod "87103603-7c26-42f0-b06d-6a980a2d7260" (UID: "87103603-7c26-42f0-b06d-6a980a2d7260"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:37:13.167404 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.167375    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87103603-7c26-42f0-b06d-6a980a2d7260-home" (OuterVolumeSpecName: "home") pod "87103603-7c26-42f0-b06d-6a980a2d7260" (UID: "87103603-7c26-42f0-b06d-6a980a2d7260"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:37:13.167624 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.167599    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87103603-7c26-42f0-b06d-6a980a2d7260-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "87103603-7c26-42f0-b06d-6a980a2d7260" (UID: "87103603-7c26-42f0-b06d-6a980a2d7260"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:37:13.168996 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.168966    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87103603-7c26-42f0-b06d-6a980a2d7260-dshm" (OuterVolumeSpecName: "dshm") pod "87103603-7c26-42f0-b06d-6a980a2d7260" (UID: "87103603-7c26-42f0-b06d-6a980a2d7260"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:37:13.169097 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.169025    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87103603-7c26-42f0-b06d-6a980a2d7260-home\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:37:13.169097 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.169047    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87103603-7c26-42f0-b06d-6a980a2d7260-cabundle-cert\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:37:13.169097 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.169063    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87103603-7c26-42f0-b06d-6a980a2d7260-model-cache\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:37:13.169403 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.169380    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87103603-7c26-42f0-b06d-6a980a2d7260-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "87103603-7c26-42f0-b06d-6a980a2d7260" (UID: "87103603-7c26-42f0-b06d-6a980a2d7260"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:37:13.169888 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.169861    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87103603-7c26-42f0-b06d-6a980a2d7260-kube-api-access-gm6r6" (OuterVolumeSpecName: "kube-api-access-gm6r6") pod "87103603-7c26-42f0-b06d-6a980a2d7260" (UID: "87103603-7c26-42f0-b06d-6a980a2d7260"). InnerVolumeSpecName "kube-api-access-gm6r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:37:13.181200 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.181174    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87103603-7c26-42f0-b06d-6a980a2d7260-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "87103603-7c26-42f0-b06d-6a980a2d7260" (UID: "87103603-7c26-42f0-b06d-6a980a2d7260"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:37:13.194417 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.194395    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87103603-7c26-42f0-b06d-6a980a2d7260-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "87103603-7c26-42f0-b06d-6a980a2d7260" (UID: "87103603-7c26-42f0-b06d-6a980a2d7260"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:37:13.269718 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.269684    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87103603-7c26-42f0-b06d-6a980a2d7260-dshm\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:37:13.269718 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.269714    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87103603-7c26-42f0-b06d-6a980a2d7260-kserve-provision-location\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:37:13.269718 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.269724    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87103603-7c26-42f0-b06d-6a980a2d7260-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:37:13.270149 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.269734    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm6r6\" (UniqueName: \"kubernetes.io/projected/87103603-7c26-42f0-b06d-6a980a2d7260-kube-api-access-gm6r6\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:37:13.270149 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.269742    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87103603-7c26-42f0-b06d-6a980a2d7260-tmp-dir\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:37:13.711813 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.711777    2583 generic.go:356] "Generic (PLEG): container finished" podID="87103603-7c26-42f0-b06d-6a980a2d7260" containerID="610d59b2c614d74e8586bf740ceba370d289000c508ea7e18a28667f5a80861e" exitCode=0
Sep 17 18:37:13.711990 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.711851    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"
Sep 17 18:37:13.711990 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.711869    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r" event={"ID":"87103603-7c26-42f0-b06d-6a980a2d7260","Type":"ContainerDied","Data":"610d59b2c614d74e8586bf740ceba370d289000c508ea7e18a28667f5a80861e"}
Sep 17 18:37:13.711990 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.711927    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r" event={"ID":"87103603-7c26-42f0-b06d-6a980a2d7260","Type":"ContainerDied","Data":"6cd33658bc38d254f8fc5e9bde3d5dae62000f5ecf7264218c60262128cb022d"}
Sep 17 18:37:13.711990 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.711952    2583 scope.go:117] "RemoveContainer" containerID="610d59b2c614d74e8586bf740ceba370d289000c508ea7e18a28667f5a80861e"
Sep 17 18:37:13.720240 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.720226    2583 scope.go:117] "RemoveContainer" containerID="3fcde1239659f093405e120eb6f085c2e4743452ed66edfb5663aecdb6e98634"
Sep 17 18:37:13.727597 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.727576    2583 scope.go:117] "RemoveContainer" containerID="610d59b2c614d74e8586bf740ceba370d289000c508ea7e18a28667f5a80861e"
Sep 17 18:37:13.727931 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:37:13.727905    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"610d59b2c614d74e8586bf740ceba370d289000c508ea7e18a28667f5a80861e\": container with ID starting with 610d59b2c614d74e8586bf740ceba370d289000c508ea7e18a28667f5a80861e not found: ID does not exist" containerID="610d59b2c614d74e8586bf740ceba370d289000c508ea7e18a28667f5a80861e"
Sep 17 18:37:13.727990 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.727944    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"610d59b2c614d74e8586bf740ceba370d289000c508ea7e18a28667f5a80861e"} err="failed to get container status \"610d59b2c614d74e8586bf740ceba370d289000c508ea7e18a28667f5a80861e\": rpc error: code = NotFound desc = could not find container \"610d59b2c614d74e8586bf740ceba370d289000c508ea7e18a28667f5a80861e\": container with ID starting with 610d59b2c614d74e8586bf740ceba370d289000c508ea7e18a28667f5a80861e not found: ID does not exist"
Sep 17 18:37:13.727990 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.727974    2583 scope.go:117] "RemoveContainer" containerID="3fcde1239659f093405e120eb6f085c2e4743452ed66edfb5663aecdb6e98634"
Sep 17 18:37:13.728256 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:37:13.728234    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fcde1239659f093405e120eb6f085c2e4743452ed66edfb5663aecdb6e98634\": container with ID starting with 3fcde1239659f093405e120eb6f085c2e4743452ed66edfb5663aecdb6e98634 not found: ID does not exist" containerID="3fcde1239659f093405e120eb6f085c2e4743452ed66edfb5663aecdb6e98634"
Sep 17 18:37:13.728318 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.728261    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fcde1239659f093405e120eb6f085c2e4743452ed66edfb5663aecdb6e98634"} err="failed to get container status \"3fcde1239659f093405e120eb6f085c2e4743452ed66edfb5663aecdb6e98634\": rpc error: code = NotFound desc = could not find container \"3fcde1239659f093405e120eb6f085c2e4743452ed66edfb5663aecdb6e98634\": container with ID starting with 3fcde1239659f093405e120eb6f085c2e4743452ed66edfb5663aecdb6e98634 not found: ID does not exist"
Sep 17 18:37:13.739156 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.739126    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"]
Sep 17 18:37:13.742380 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:13.742354    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-5dc9b5b887-4276r"]
Sep 17 18:37:14.439209 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:14.439165    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2" podUID="f1c90212-326c-452f-9831-e8a4a13936e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 17 18:37:15.040894 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:15.040856    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87103603-7c26-42f0-b06d-6a980a2d7260" path="/var/lib/kubelet/pods/87103603-7c26-42f0-b06d-6a980a2d7260/volumes"
Sep 17 18:37:24.438765 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:24.438718    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2" podUID="f1c90212-326c-452f-9831-e8a4a13936e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 17 18:37:34.439217 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:34.439177    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2" podUID="f1c90212-326c-452f-9831-e8a4a13936e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 17 18:37:44.449299 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:44.449260    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:37:44.457066 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:37:44.457043    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:38:09.384398 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:09.384363    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-7c7446d756-568f2_f1c90212-326c-452f-9831-e8a4a13936e9/storage-initializer/0.log"
Sep 17 18:38:09.393456 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:09.393433    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-7c7446d756-568f2_f1c90212-326c-452f-9831-e8a4a13936e9/main/0.log"
Sep 17 18:38:27.933557 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:27.933523    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"]
Sep 17 18:38:42.955392 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:42.955344    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2" podUID="f1c90212-326c-452f-9831-e8a4a13936e9" containerName="main" containerID="cri-o://adb2b07a80df337b056461a1b4e63cf474d3afe96f680a001a6148dca7f99146" gracePeriod=45
Sep 17 18:38:43.449895 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:43.449849    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2" podUID="f1c90212-326c-452f-9831-e8a4a13936e9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 17 18:38:43.605613 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:43.605580    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:38:43.698167 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:43.698089    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1c90212-326c-452f-9831-e8a4a13936e9-cabundle-cert\") pod \"f1c90212-326c-452f-9831-e8a4a13936e9\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") "
Sep 17 18:38:43.698167 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:43.698137    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1c90212-326c-452f-9831-e8a4a13936e9-tls-certs\") pod \"f1c90212-326c-452f-9831-e8a4a13936e9\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") "
Sep 17 18:38:43.698356 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:43.698172    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1c90212-326c-452f-9831-e8a4a13936e9-kserve-provision-location\") pod \"f1c90212-326c-452f-9831-e8a4a13936e9\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") "
Sep 17 18:38:43.698356 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:43.698275    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"lora-pvc-lora-adapter-1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05043ab4858da5f0e\") pod \"f1c90212-326c-452f-9831-e8a4a13936e9\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") "
Sep 17 18:38:43.698356 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:43.698323    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1c90212-326c-452f-9831-e8a4a13936e9-model-cache\") pod \"f1c90212-326c-452f-9831-e8a4a13936e9\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") "
Sep 17 18:38:43.698356 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:43.698347    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1c90212-326c-452f-9831-e8a4a13936e9-dshm\") pod \"f1c90212-326c-452f-9831-e8a4a13936e9\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") "
Sep 17 18:38:43.698565 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:43.698398    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6pbg\" (UniqueName: \"kubernetes.io/projected/f1c90212-326c-452f-9831-e8a4a13936e9-kube-api-access-x6pbg\") pod \"f1c90212-326c-452f-9831-e8a4a13936e9\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") "
Sep 17 18:38:43.698565 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:43.698428    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1c90212-326c-452f-9831-e8a4a13936e9-tmp-dir\") pod \"f1c90212-326c-452f-9831-e8a4a13936e9\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") "
Sep 17 18:38:43.698565 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:43.698452    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1c90212-326c-452f-9831-e8a4a13936e9-home\") pod \"f1c90212-326c-452f-9831-e8a4a13936e9\" (UID: \"f1c90212-326c-452f-9831-e8a4a13936e9\") "
Sep 17 18:38:43.698770 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:43.698508    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1c90212-326c-452f-9831-e8a4a13936e9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f1c90212-326c-452f-9831-e8a4a13936e9" (UID: "f1c90212-326c-452f-9831-e8a4a13936e9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:38:43.698770 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:43.698740    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f1c90212-326c-452f-9831-e8a4a13936e9-cabundle-cert\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:38:43.698877 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:43.698858    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1c90212-326c-452f-9831-e8a4a13936e9-model-cache" (OuterVolumeSpecName: "model-cache") pod "f1c90212-326c-452f-9831-e8a4a13936e9" (UID: "f1c90212-326c-452f-9831-e8a4a13936e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:38:43.699073 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:43.699020    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1c90212-326c-452f-9831-e8a4a13936e9-home" (OuterVolumeSpecName: "home") pod "f1c90212-326c-452f-9831-e8a4a13936e9" (UID: "f1c90212-326c-452f-9831-e8a4a13936e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:38:43.700736 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:43.700619    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1c90212-326c-452f-9831-e8a4a13936e9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f1c90212-326c-452f-9831-e8a4a13936e9" (UID: "f1c90212-326c-452f-9831-e8a4a13936e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:38:43.700831 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:43.700784    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1c90212-326c-452f-9831-e8a4a13936e9-dshm" (OuterVolumeSpecName: "dshm") pod "f1c90212-326c-452f-9831-e8a4a13936e9" (UID: "f1c90212-326c-452f-9831-e8a4a13936e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:38:43.701056 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:43.701027    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1c90212-326c-452f-9831-e8a4a13936e9-kube-api-access-x6pbg" (OuterVolumeSpecName: "kube-api-access-x6pbg") pod "f1c90212-326c-452f-9831-e8a4a13936e9" (UID: "f1c90212-326c-452f-9831-e8a4a13936e9"). InnerVolumeSpecName "kube-api-access-x6pbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:38:43.703623 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:43.703601    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-05043ab4858da5f0e" (OuterVolumeSpecName: "lora-pvc-lora-adapter-1") pod "f1c90212-326c-452f-9831-e8a4a13936e9" (UID: "f1c90212-326c-452f-9831-e8a4a13936e9"). InnerVolumeSpecName "pvc-048a3c21-2fe1-48d8-9c35-8c255bd034d8". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 18:38:43.717162 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:43.717137    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1c90212-326c-452f-9831-e8a4a13936e9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f1c90212-326c-452f-9831-e8a4a13936e9" (UID: "f1c90212-326c-452f-9831-e8a4a13936e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:38:43.727634 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:43.727611    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1c90212-326c-452f-9831-e8a4a13936e9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f1c90212-326c-452f-9831-e8a4a13936e9" (UID: "f1c90212-326c-452f-9831-e8a4a13936e9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:38:43.799974 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:43.799953    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1c90212-326c-452f-9831-e8a4a13936e9-home\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:38:43.799974 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:43.799973    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1c90212-326c-452f-9831-e8a4a13936e9-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:38:43.800098 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:43.799985    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1c90212-326c-452f-9831-e8a4a13936e9-kserve-provision-location\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:38:43.800098 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:43.800014    2583 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-048a3c21-2fe1-48d8-9c35-8c255bd034d8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05043ab4858da5f0e\") on node \"ip-10-0-141-30.ec2.internal\" "
Sep 17 18:38:43.800098 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:43.800024    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1c90212-326c-452f-9831-e8a4a13936e9-model-cache\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:38:43.800098 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:43.800032    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1c90212-326c-452f-9831-e8a4a13936e9-dshm\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:38:43.800098 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:43.800039    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6pbg\" (UniqueName: \"kubernetes.io/projected/f1c90212-326c-452f-9831-e8a4a13936e9-kube-api-access-x6pbg\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:38:43.800098 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:43.800047    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1c90212-326c-452f-9831-e8a4a13936e9-tmp-dir\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:38:43.810543 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:43.810528    2583 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-048a3c21-2fe1-48d8-9c35-8c255bd034d8" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-05043ab4858da5f0e") on node "ip-10-0-141-30.ec2.internal"
Sep 17 18:38:43.900944 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:43.900912    2583 reconciler_common.go:299] "Volume detached for volume \"pvc-048a3c21-2fe1-48d8-9c35-8c255bd034d8\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-05043ab4858da5f0e\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"csi-964e7867a9e2c3c7a0c2e42abe77980f25c3fb763de53291fa381acca5db41f5\""
Sep 17 18:38:44.112302 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:44.112272    2583 generic.go:356] "Generic (PLEG): container finished" podID="f1c90212-326c-452f-9831-e8a4a13936e9" containerID="adb2b07a80df337b056461a1b4e63cf474d3afe96f680a001a6148dca7f99146" exitCode=0
Sep 17 18:38:44.112828 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:44.112320    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"
Sep 17 18:38:44.112828 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:44.112346    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2" event={"ID":"f1c90212-326c-452f-9831-e8a4a13936e9","Type":"ContainerDied","Data":"adb2b07a80df337b056461a1b4e63cf474d3afe96f680a001a6148dca7f99146"}
Sep 17 18:38:44.112828 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:44.112398    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2" event={"ID":"f1c90212-326c-452f-9831-e8a4a13936e9","Type":"ContainerDied","Data":"cd7b66e31c2879b1108ece0ed90e4f0675fd11d2affedb58a4e607e0eff39235"}
Sep 17 18:38:44.112828 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:44.112430    2583 scope.go:117] "RemoveContainer" containerID="adb2b07a80df337b056461a1b4e63cf474d3afe96f680a001a6148dca7f99146"
Sep 17 18:38:44.121208 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:44.121190    2583 scope.go:117] "RemoveContainer" containerID="0b78d4abdaaaab1b558ff43eb6dc46a2e3cb6920ed0bed91cf37193313d8d28a"
Sep 17 18:38:44.128365 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:44.128347    2583 scope.go:117] "RemoveContainer" containerID="adb2b07a80df337b056461a1b4e63cf474d3afe96f680a001a6148dca7f99146"
Sep 17 18:38:44.128595 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:38:44.128573    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adb2b07a80df337b056461a1b4e63cf474d3afe96f680a001a6148dca7f99146\": container with ID starting with adb2b07a80df337b056461a1b4e63cf474d3afe96f680a001a6148dca7f99146 not found: ID does not exist" containerID="adb2b07a80df337b056461a1b4e63cf474d3afe96f680a001a6148dca7f99146"
Sep 17 18:38:44.128689 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:44.128604    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adb2b07a80df337b056461a1b4e63cf474d3afe96f680a001a6148dca7f99146"} err="failed to get container status \"adb2b07a80df337b056461a1b4e63cf474d3afe96f680a001a6148dca7f99146\": rpc error: code = NotFound desc = could not find container \"adb2b07a80df337b056461a1b4e63cf474d3afe96f680a001a6148dca7f99146\": container with ID starting with adb2b07a80df337b056461a1b4e63cf474d3afe96f680a001a6148dca7f99146 not found: ID does not exist"
Sep 17 18:38:44.128689 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:44.128629    2583 scope.go:117] "RemoveContainer" containerID="0b78d4abdaaaab1b558ff43eb6dc46a2e3cb6920ed0bed91cf37193313d8d28a"
Sep 17 18:38:44.128899 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:38:44.128881    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b78d4abdaaaab1b558ff43eb6dc46a2e3cb6920ed0bed91cf37193313d8d28a\": container with ID starting with 0b78d4abdaaaab1b558ff43eb6dc46a2e3cb6920ed0bed91cf37193313d8d28a not found: ID does not exist" containerID="0b78d4abdaaaab1b558ff43eb6dc46a2e3cb6920ed0bed91cf37193313d8d28a"
Sep 17 18:38:44.128947 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:44.128905    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b78d4abdaaaab1b558ff43eb6dc46a2e3cb6920ed0bed91cf37193313d8d28a"} err="failed to get container status \"0b78d4abdaaaab1b558ff43eb6dc46a2e3cb6920ed0bed91cf37193313d8d28a\": rpc error: code = NotFound desc = could not find container \"0b78d4abdaaaab1b558ff43eb6dc46a2e3cb6920ed0bed91cf37193313d8d28a\": container with ID starting with 0b78d4abdaaaab1b558ff43eb6dc46a2e3cb6920ed0bed91cf37193313d8d28a not found: ID does not exist"
Sep 17 18:38:44.133827 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:44.133806    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"]
Sep 17 18:38:44.137937 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:44.137917    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-7c7446d756-568f2"]
Sep 17 18:38:45.040190 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:45.040155    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1c90212-326c-452f-9831-e8a4a13936e9" path="/var/lib/kubelet/pods/f1c90212-326c-452f-9831-e8a4a13936e9/volumes"
Sep 17 18:38:49.141713 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:49.141680    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8jf4l"]
Sep 17 18:38:49.142108 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:49.141952    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87103603-7c26-42f0-b06d-6a980a2d7260" containerName="storage-initializer"
Sep 17 18:38:49.142108 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:49.141962    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="87103603-7c26-42f0-b06d-6a980a2d7260" containerName="storage-initializer"
Sep 17 18:38:49.142108 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:49.141973    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1c90212-326c-452f-9831-e8a4a13936e9" containerName="storage-initializer"
Sep 17 18:38:49.142108 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:49.141979    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1c90212-326c-452f-9831-e8a4a13936e9" containerName="storage-initializer"
Sep 17 18:38:49.142108 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:49.141985    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1c90212-326c-452f-9831-e8a4a13936e9" containerName="main"
Sep 17 18:38:49.142108 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:49.141989    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1c90212-326c-452f-9831-e8a4a13936e9" containerName="main"
Sep 17 18:38:49.142108 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:49.142002    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87103603-7c26-42f0-b06d-6a980a2d7260" containerName="main"
Sep 17 18:38:49.142108 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:49.142007    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="87103603-7c26-42f0-b06d-6a980a2d7260" containerName="main"
Sep 17 18:38:49.142108 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:49.142054    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1c90212-326c-452f-9831-e8a4a13936e9" containerName="main"
Sep 17 18:38:49.142108 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:49.142063    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="87103603-7c26-42f0-b06d-6a980a2d7260" containerName="main"
Sep 17 18:38:49.145690 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:49.145652    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8jf4l"
Sep 17 18:38:49.148291 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:49.148272    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 17 18:38:49.148404 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:49.148306    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 17 18:38:49.148556 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:49.148539    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"seaweedfs-s3-creds\""
Sep 17 18:38:49.149133 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:49.149119    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-ml45c\""
Sep 17 18:38:49.154113 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:49.154090    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8jf4l"]
Sep 17 18:38:49.244238 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:49.244209    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h727\" (UniqueName: \"kubernetes.io/projected/162810fc-3ab1-470a-985b-0df31ca2710a-kube-api-access-2h727\") pod \"e2e-pvc-lora-download-8jf4l\" (UID: \"162810fc-3ab1-470a-985b-0df31ca2710a\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8jf4l"
Sep 17 18:38:49.344962 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:49.344928    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h727\" (UniqueName: \"kubernetes.io/projected/162810fc-3ab1-470a-985b-0df31ca2710a-kube-api-access-2h727\") pod \"e2e-pvc-lora-download-8jf4l\" (UID: \"162810fc-3ab1-470a-985b-0df31ca2710a\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8jf4l"
Sep 17 18:38:49.354070 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:49.354041    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h727\" (UniqueName: \"kubernetes.io/projected/162810fc-3ab1-470a-985b-0df31ca2710a-kube-api-access-2h727\") pod \"e2e-pvc-lora-download-8jf4l\" (UID: \"162810fc-3ab1-470a-985b-0df31ca2710a\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8jf4l"
Sep 17 18:38:50.801235 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:50.801203    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-tvmm2"]
Sep 17 18:38:50.804433 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:50.804412    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-tvmm2"]
Sep 17 18:38:51.040401 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:51.040325    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa5f1843-76b6-42c5-8afb-2e00fefeff4e" path="/var/lib/kubelet/pods/aa5f1843-76b6-42c5-8afb-2e00fefeff4e/volumes"
Sep 17 18:38:57.008338 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:57.008305    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-343bb4b6-63b9-4a43-94ea-e0086891b24e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-013c3c644f71dc3fe\") pod \"e2e-pvc-lora-download-8jf4l\" (UID: \"162810fc-3ab1-470a-985b-0df31ca2710a\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8jf4l"
Sep 17 18:38:57.013779 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:57.013756    2583 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-343bb4b6-63b9-4a43-94ea-e0086891b24e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-013c3c644f71dc3fe\") pod \"e2e-pvc-lora-download-8jf4l\" (UID: \"162810fc-3ab1-470a-985b-0df31ca2710a\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8jf4l"
Sep 17 18:38:57.108707 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:57.108681    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-343bb4b6-63b9-4a43-94ea-e0086891b24e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-013c3c644f71dc3fe\") pod \"e2e-pvc-lora-download-8jf4l\" (UID: \"162810fc-3ab1-470a-985b-0df31ca2710a\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8jf4l"
Sep 17 18:38:57.108863 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:57.108803    2583 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-343bb4b6-63b9-4a43-94ea-e0086891b24e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-013c3c644f71dc3fe\") pod \"e2e-pvc-lora-download-8jf4l\" (UID: \"162810fc-3ab1-470a-985b-0df31ca2710a\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8jf4l"
Sep 17 18:38:57.112843 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:57.112823    2583 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-343bb4b6-63b9-4a43-94ea-e0086891b24e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-013c3c644f71dc3fe\") pod \"e2e-pvc-lora-download-8jf4l\" (UID: \"162810fc-3ab1-470a-985b-0df31ca2710a\") DevicePath \"csi-9f7da773de440f6236adb22edf2326011dd872147e48a3dd598142d05d5e845d\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8jf4l"
Sep 17 18:38:57.323787 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:57.323759    2583 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-343bb4b6-63b9-4a43-94ea-e0086891b24e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-013c3c644f71dc3fe\") pod \"e2e-pvc-lora-download-8jf4l\" (UID: \"162810fc-3ab1-470a-985b-0df31ca2710a\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/66f33fb4ae41959ba0894d927c823da785849effc116e74f6244a4cb88c5813f/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8jf4l"
Sep 17 18:38:57.336158 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:57.336136    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-343bb4b6-63b9-4a43-94ea-e0086891b24e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-013c3c644f71dc3fe\") pod \"e2e-pvc-lora-download-8jf4l\" (UID: \"162810fc-3ab1-470a-985b-0df31ca2710a\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8jf4l"
Sep 17 18:38:57.557825 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:57.557788    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-ml45c\""
Sep 17 18:38:57.565794 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:57.565768    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8jf4l"
Sep 17 18:38:57.834166 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:57.833998    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8jf4l"]
Sep 17 18:38:57.836718 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:38:57.836690    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod162810fc_3ab1_470a_985b_0df31ca2710a.slice/crio-86a195840273e8624aaca68d1c92add9fa04d512fe971f2244e07da18bb7607d WatchSource:0}: Error finding container 86a195840273e8624aaca68d1c92add9fa04d512fe971f2244e07da18bb7607d: Status 404 returned error can't find the container with id 86a195840273e8624aaca68d1c92add9fa04d512fe971f2244e07da18bb7607d
Sep 17 18:38:57.840718 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:57.840651    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"seaweedfs-s3-creds\""
Sep 17 18:38:58.173613 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:58.173579    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8jf4l" event={"ID":"162810fc-3ab1-470a-985b-0df31ca2710a","Type":"ContainerStarted","Data":"c0729f852a07cb8cb9010bb8140875e17065302b6aea62cbee20bc628f073c4b"}
Sep 17 18:38:58.174107 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:58.173625    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8jf4l" event={"ID":"162810fc-3ab1-470a-985b-0df31ca2710a","Type":"ContainerStarted","Data":"86a195840273e8624aaca68d1c92add9fa04d512fe971f2244e07da18bb7607d"}
Sep 17 18:38:58.189731 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:38:58.189686    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8jf4l" podStartSLOduration=12.189653921 podStartE2EDuration="12.189653921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:38:58.188758056 +0000 UTC m=+3849.280064822" watchObservedRunningTime="2026-09-17 18:38:58.189653921 +0000 UTC m=+3849.280960686"
Sep 17 18:39:00.183691 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:39:00.183638    2583 generic.go:356] "Generic (PLEG): container finished" podID="162810fc-3ab1-470a-985b-0df31ca2710a" containerID="c0729f852a07cb8cb9010bb8140875e17065302b6aea62cbee20bc628f073c4b" exitCode=0
Sep 17 18:39:00.184096 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:39:00.183703    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8jf4l" event={"ID":"162810fc-3ab1-470a-985b-0df31ca2710a","Type":"ContainerDied","Data":"c0729f852a07cb8cb9010bb8140875e17065302b6aea62cbee20bc628f073c4b"}
Sep 17 18:39:01.322596 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:39:01.322565    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8jf4l"
Sep 17 18:39:01.444308 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:39:01.444230    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-013c3c644f71dc3fe\") pod \"162810fc-3ab1-470a-985b-0df31ca2710a\" (UID: \"162810fc-3ab1-470a-985b-0df31ca2710a\") "
Sep 17 18:39:01.444308 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:39:01.444286    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2h727\" (UniqueName: \"kubernetes.io/projected/162810fc-3ab1-470a-985b-0df31ca2710a-kube-api-access-2h727\") pod \"162810fc-3ab1-470a-985b-0df31ca2710a\" (UID: \"162810fc-3ab1-470a-985b-0df31ca2710a\") "
Sep 17 18:39:01.446553 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:39:01.446513    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/162810fc-3ab1-470a-985b-0df31ca2710a-kube-api-access-2h727" (OuterVolumeSpecName: "kube-api-access-2h727") pod "162810fc-3ab1-470a-985b-0df31ca2710a" (UID: "162810fc-3ab1-470a-985b-0df31ca2710a"). InnerVolumeSpecName "kube-api-access-2h727". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:39:01.446957 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:39:01.446934    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-013c3c644f71dc3fe" (OuterVolumeSpecName: "model-storage") pod "162810fc-3ab1-470a-985b-0df31ca2710a" (UID: "162810fc-3ab1-470a-985b-0df31ca2710a"). InnerVolumeSpecName "pvc-343bb4b6-63b9-4a43-94ea-e0086891b24e". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 17 18:39:01.545415 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:39:01.545367    2583 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-343bb4b6-63b9-4a43-94ea-e0086891b24e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-013c3c644f71dc3fe\") on node \"ip-10-0-141-30.ec2.internal\" "
Sep 17 18:39:01.545415 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:39:01.545409    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h727\" (UniqueName: \"kubernetes.io/projected/162810fc-3ab1-470a-985b-0df31ca2710a-kube-api-access-2h727\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:39:02.192253 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:39:02.192166    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8jf4l"
Sep 17 18:39:02.192253 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:39:02.192186    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8jf4l" event={"ID":"162810fc-3ab1-470a-985b-0df31ca2710a","Type":"ContainerDied","Data":"86a195840273e8624aaca68d1c92add9fa04d512fe971f2244e07da18bb7607d"}
Sep 17 18:39:02.192253 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:39:02.192225    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86a195840273e8624aaca68d1c92add9fa04d512fe971f2244e07da18bb7607d"
Sep 17 18:39:02.517415 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:39:02.517392    2583 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-343bb4b6-63b9-4a43-94ea-e0086891b24e" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-013c3c644f71dc3fe") on node "ip-10-0-141-30.ec2.internal"
Sep 17 18:39:02.552791 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:39:02.552761    2583 reconciler_common.go:299] "Volume detached for volume \"pvc-343bb4b6-63b9-4a43-94ea-e0086891b24e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-013c3c644f71dc3fe\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"csi-9f7da773de440f6236adb22edf2326011dd872147e48a3dd598142d05d5e845d\""
Sep 17 18:40:34.781581 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:34.781547    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj"]
Sep 17 18:40:34.782052 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:34.781852    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="162810fc-3ab1-470a-985b-0df31ca2710a" containerName="storage-initializer"
Sep 17 18:40:34.782052 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:34.781863    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="162810fc-3ab1-470a-985b-0df31ca2710a" containerName="storage-initializer"
Sep 17 18:40:34.782052 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:34.781915    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="162810fc-3ab1-470a-985b-0df31ca2710a" containerName="storage-initializer"
Sep 17 18:40:34.784610 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:34.784593    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj"
Sep 17 18:40:34.787999 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:34.787977    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-t59sq\""
Sep 17 18:40:34.788193 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:34.788091    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-wfdbf\""
Sep 17 18:40:34.788288 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:34.788091    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 17 18:40:34.788351 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:34.788137    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 17 18:40:34.788351 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:34.788140    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 17 18:40:34.798068 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:34.798047    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj"]
Sep 17 18:40:34.852198 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:34.852161    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 18:40:34.853809 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:34.853782    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpg97\" (UniqueName: \"kubernetes.io/projected/48352ae8-12dd-4a0e-a30c-ea2e3e5639a3-kube-api-access-xpg97\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj\" (UID: \"48352ae8-12dd-4a0e-a30c-ea2e3e5639a3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj"
Sep 17 18:40:34.853950 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:34.853831    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48352ae8-12dd-4a0e-a30c-ea2e3e5639a3-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj\" (UID: \"48352ae8-12dd-4a0e-a30c-ea2e3e5639a3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj"
Sep 17 18:40:34.855470 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:34.855455    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:40:34.857878 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:34.857856    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 17 18:40:34.857991 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:34.857856    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-tpc9j\""
Sep 17 18:40:34.857991 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:34.857857    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:40:34.866926 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:34.866906    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 18:40:34.954529 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:34.954499    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpg97\" (UniqueName: \"kubernetes.io/projected/48352ae8-12dd-4a0e-a30c-ea2e3e5639a3-kube-api-access-xpg97\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj\" (UID: \"48352ae8-12dd-4a0e-a30c-ea2e3e5639a3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj"
Sep 17 18:40:34.954745 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:34.954554    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2762161a-5261-4ece-b75a-5a6d8e0c1878-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2762161a-5261-4ece-b75a-5a6d8e0c1878\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:40:34.954745 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:34.954589    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2762161a-5261-4ece-b75a-5a6d8e0c1878-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2762161a-5261-4ece-b75a-5a6d8e0c1878\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:40:34.954745 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:34.954617    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48352ae8-12dd-4a0e-a30c-ea2e3e5639a3-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj\" (UID: \"48352ae8-12dd-4a0e-a30c-ea2e3e5639a3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj"
Sep 17 18:40:34.954745 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:34.954640    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2762161a-5261-4ece-b75a-5a6d8e0c1878-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2762161a-5261-4ece-b75a-5a6d8e0c1878\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:40:34.954745 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:34.954693    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2762161a-5261-4ece-b75a-5a6d8e0c1878-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2762161a-5261-4ece-b75a-5a6d8e0c1878\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:40:34.954986 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:34.954802    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2762161a-5261-4ece-b75a-5a6d8e0c1878-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2762161a-5261-4ece-b75a-5a6d8e0c1878\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:40:34.954986 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:34.954853    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppkfh\" (UniqueName: \"kubernetes.io/projected/2762161a-5261-4ece-b75a-5a6d8e0c1878-kube-api-access-ppkfh\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2762161a-5261-4ece-b75a-5a6d8e0c1878\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:40:34.954986 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:34.954891    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2762161a-5261-4ece-b75a-5a6d8e0c1878-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2762161a-5261-4ece-b75a-5a6d8e0c1878\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:40:34.954986 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:34.954928    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2762161a-5261-4ece-b75a-5a6d8e0c1878-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2762161a-5261-4ece-b75a-5a6d8e0c1878\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:40:34.957190 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:34.957159    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48352ae8-12dd-4a0e-a30c-ea2e3e5639a3-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj\" (UID: \"48352ae8-12dd-4a0e-a30c-ea2e3e5639a3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj"
Sep 17 18:40:34.962188 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:34.962157    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpg97\" (UniqueName: \"kubernetes.io/projected/48352ae8-12dd-4a0e-a30c-ea2e3e5639a3-kube-api-access-xpg97\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj\" (UID: \"48352ae8-12dd-4a0e-a30c-ea2e3e5639a3\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj"
Sep 17 18:40:35.055370 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:35.055294    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2762161a-5261-4ece-b75a-5a6d8e0c1878-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2762161a-5261-4ece-b75a-5a6d8e0c1878\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:40:35.055370 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:35.055340    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2762161a-5261-4ece-b75a-5a6d8e0c1878-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2762161a-5261-4ece-b75a-5a6d8e0c1878\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:40:35.055747 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:35.055375    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2762161a-5261-4ece-b75a-5a6d8e0c1878-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2762161a-5261-4ece-b75a-5a6d8e0c1878\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:40:35.055747 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:35.055396    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2762161a-5261-4ece-b75a-5a6d8e0c1878-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2762161a-5261-4ece-b75a-5a6d8e0c1878\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:40:35.055747 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:35.055424    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2762161a-5261-4ece-b75a-5a6d8e0c1878-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2762161a-5261-4ece-b75a-5a6d8e0c1878\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:40:35.055747 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:35.055442    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2762161a-5261-4ece-b75a-5a6d8e0c1878-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2762161a-5261-4ece-b75a-5a6d8e0c1878\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:40:35.055747 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:35.055469    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2762161a-5261-4ece-b75a-5a6d8e0c1878-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2762161a-5261-4ece-b75a-5a6d8e0c1878\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:40:35.055747 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:35.055497    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppkfh\" (UniqueName: \"kubernetes.io/projected/2762161a-5261-4ece-b75a-5a6d8e0c1878-kube-api-access-ppkfh\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2762161a-5261-4ece-b75a-5a6d8e0c1878\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:40:35.056113 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:35.055760    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2762161a-5261-4ece-b75a-5a6d8e0c1878-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2762161a-5261-4ece-b75a-5a6d8e0c1878\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:40:35.056113 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:35.055813    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2762161a-5261-4ece-b75a-5a6d8e0c1878-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2762161a-5261-4ece-b75a-5a6d8e0c1878\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:40:35.056113 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:35.056010    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2762161a-5261-4ece-b75a-5a6d8e0c1878-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2762161a-5261-4ece-b75a-5a6d8e0c1878\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:40:35.056113 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:35.056040    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2762161a-5261-4ece-b75a-5a6d8e0c1878-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2762161a-5261-4ece-b75a-5a6d8e0c1878\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:40:35.056113 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:35.056048    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2762161a-5261-4ece-b75a-5a6d8e0c1878-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2762161a-5261-4ece-b75a-5a6d8e0c1878\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:40:35.057596 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:35.057573    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2762161a-5261-4ece-b75a-5a6d8e0c1878-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2762161a-5261-4ece-b75a-5a6d8e0c1878\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:40:35.057876 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:35.057860    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2762161a-5261-4ece-b75a-5a6d8e0c1878-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2762161a-5261-4ece-b75a-5a6d8e0c1878\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:40:35.062953 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:35.062931    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppkfh\" (UniqueName: \"kubernetes.io/projected/2762161a-5261-4ece-b75a-5a6d8e0c1878-kube-api-access-ppkfh\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2762161a-5261-4ece-b75a-5a6d8e0c1878\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:40:35.096884 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:35.096864    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj"
Sep 17 18:40:35.165267 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:35.165239    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:40:35.366536 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:35.366510    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj"]
Sep 17 18:40:35.369162 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:40:35.369130    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48352ae8_12dd_4a0e_a30c_ea2e3e5639a3.slice/crio-fc6855578b125b28f5da684f846734d481c68de6e453d5ffdaadcfecb895ec07 WatchSource:0}: Error finding container fc6855578b125b28f5da684f846734d481c68de6e453d5ffdaadcfecb895ec07: Status 404 returned error can't find the container with id fc6855578b125b28f5da684f846734d481c68de6e453d5ffdaadcfecb895ec07
Sep 17 18:40:35.439748 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:35.439720    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 18:40:35.442759 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:40:35.442725    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2762161a_5261_4ece_b75a_5a6d8e0c1878.slice/crio-e6789ea1faa57567bb84d111ce369f02ae9a28ea3cce523e4bee200ccf0d0fd8 WatchSource:0}: Error finding container e6789ea1faa57567bb84d111ce369f02ae9a28ea3cce523e4bee200ccf0d0fd8: Status 404 returned error can't find the container with id e6789ea1faa57567bb84d111ce369f02ae9a28ea3cce523e4bee200ccf0d0fd8
Sep 17 18:40:35.582858 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:35.582814    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj" event={"ID":"48352ae8-12dd-4a0e-a30c-ea2e3e5639a3","Type":"ContainerStarted","Data":"8d462a27a9bb885874c6fa4c20323de555404814bad05562ab30b826f6db1f16"}
Sep 17 18:40:35.583049 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:35.582886    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj"
Sep 17 18:40:35.583049 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:35.582913    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj" event={"ID":"48352ae8-12dd-4a0e-a30c-ea2e3e5639a3","Type":"ContainerStarted","Data":"fc6855578b125b28f5da684f846734d481c68de6e453d5ffdaadcfecb895ec07"}
Sep 17 18:40:35.584525 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:35.584498    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"2762161a-5261-4ece-b75a-5a6d8e0c1878","Type":"ContainerStarted","Data":"f3f329dde7b0ed200a8edde0feceacf61b1107860e09f25412c7ba879d67b5b4"}
Sep 17 18:40:35.584633 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:35.584540    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"2762161a-5261-4ece-b75a-5a6d8e0c1878","Type":"ContainerStarted","Data":"e6789ea1faa57567bb84d111ce369f02ae9a28ea3cce523e4bee200ccf0d0fd8"}
Sep 17 18:40:35.602770 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:35.602685    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj" podStartSLOduration=1.602645683 podStartE2EDuration="1.602645683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:35.601329388 +0000 UTC m=+3946.692636154" watchObservedRunningTime="2026-09-17 18:40:35.602645683 +0000 UTC m=+3946.693952464"
Sep 17 18:40:38.598608 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:38.598574    2583 generic.go:356] "Generic (PLEG): container finished" podID="2762161a-5261-4ece-b75a-5a6d8e0c1878" containerID="f3f329dde7b0ed200a8edde0feceacf61b1107860e09f25412c7ba879d67b5b4" exitCode=0
Sep 17 18:40:38.598608 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:38.598613    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"2762161a-5261-4ece-b75a-5a6d8e0c1878","Type":"ContainerDied","Data":"f3f329dde7b0ed200a8edde0feceacf61b1107860e09f25412c7ba879d67b5b4"}
Sep 17 18:40:39.604929 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:39.604893    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"2762161a-5261-4ece-b75a-5a6d8e0c1878","Type":"ContainerStarted","Data":"b59a35c467476e829dc0ac0c301022c8aa55dd008d1372c78ea0adb52b3959ea"}
Sep 17 18:40:39.629675 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:40:39.629605    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=3.513884017 podStartE2EDuration="5.629583387s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.11569937s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:39.628292374 +0000 UTC m=+3950.719599158" watchObservedRunningTime="2026-09-17 18:40:39.629583387 +0000 UTC m=+3950.720890153"
Sep 17 18:41:06.591347 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:06.591302    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj"
Sep 17 18:41:38.903215 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:38.903177    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"]
Sep 17 18:41:38.906736 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:38.906717    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"
Sep 17 18:41:38.909209 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:38.909189    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Sep 17 18:41:38.909334 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:38.909234    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-v4scn\""
Sep 17 18:41:38.909520 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:38.909497    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 17 18:41:38.910256 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:38.910227    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:41:38.910407 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:38.910229    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 17 18:41:38.910407 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:38.910351    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 17 18:41:38.920100 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:38.920078    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"]
Sep 17 18:41:38.985710 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:38.985674    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/920d484e-62d1-4a98-aa61-b5669584dac5-tls-certs\") pod \"tls-verification-test-kserve-85c4d7bc5b-pbqpl\" (UID: \"920d484e-62d1-4a98-aa61-b5669584dac5\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"
Sep 17 18:41:38.985710 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:38.985711    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/920d484e-62d1-4a98-aa61-b5669584dac5-dshm\") pod \"tls-verification-test-kserve-85c4d7bc5b-pbqpl\" (UID: \"920d484e-62d1-4a98-aa61-b5669584dac5\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"
Sep 17 18:41:38.985942 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:38.985819    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/920d484e-62d1-4a98-aa61-b5669584dac5-home\") pod \"tls-verification-test-kserve-85c4d7bc5b-pbqpl\" (UID: \"920d484e-62d1-4a98-aa61-b5669584dac5\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"
Sep 17 18:41:38.985942 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:38.985873    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/920d484e-62d1-4a98-aa61-b5669584dac5-cabundle-cert\") pod \"tls-verification-test-kserve-85c4d7bc5b-pbqpl\" (UID: \"920d484e-62d1-4a98-aa61-b5669584dac5\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"
Sep 17 18:41:38.985942 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:38.985903    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz7cb\" (UniqueName: \"kubernetes.io/projected/920d484e-62d1-4a98-aa61-b5669584dac5-kube-api-access-qz7cb\") pod \"tls-verification-test-kserve-85c4d7bc5b-pbqpl\" (UID: \"920d484e-62d1-4a98-aa61-b5669584dac5\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"
Sep 17 18:41:38.986051 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:38.985972    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920d484e-62d1-4a98-aa61-b5669584dac5-tmp-dir\") pod \"tls-verification-test-kserve-85c4d7bc5b-pbqpl\" (UID: \"920d484e-62d1-4a98-aa61-b5669584dac5\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"
Sep 17 18:41:38.986051 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:38.986000    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/920d484e-62d1-4a98-aa61-b5669584dac5-kserve-provision-location\") pod \"tls-verification-test-kserve-85c4d7bc5b-pbqpl\" (UID: \"920d484e-62d1-4a98-aa61-b5669584dac5\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"
Sep 17 18:41:38.986051 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:38.986018    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920d484e-62d1-4a98-aa61-b5669584dac5-model-cache\") pod \"tls-verification-test-kserve-85c4d7bc5b-pbqpl\" (UID: \"920d484e-62d1-4a98-aa61-b5669584dac5\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"
Sep 17 18:41:39.087179 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:39.087140    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/920d484e-62d1-4a98-aa61-b5669584dac5-home\") pod \"tls-verification-test-kserve-85c4d7bc5b-pbqpl\" (UID: \"920d484e-62d1-4a98-aa61-b5669584dac5\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"
Sep 17 18:41:39.087392 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:39.087186    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/920d484e-62d1-4a98-aa61-b5669584dac5-cabundle-cert\") pod \"tls-verification-test-kserve-85c4d7bc5b-pbqpl\" (UID: \"920d484e-62d1-4a98-aa61-b5669584dac5\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"
Sep 17 18:41:39.087392 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:39.087329    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz7cb\" (UniqueName: \"kubernetes.io/projected/920d484e-62d1-4a98-aa61-b5669584dac5-kube-api-access-qz7cb\") pod \"tls-verification-test-kserve-85c4d7bc5b-pbqpl\" (UID: \"920d484e-62d1-4a98-aa61-b5669584dac5\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"
Sep 17 18:41:39.087532 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:39.087448    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920d484e-62d1-4a98-aa61-b5669584dac5-tmp-dir\") pod \"tls-verification-test-kserve-85c4d7bc5b-pbqpl\" (UID: \"920d484e-62d1-4a98-aa61-b5669584dac5\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"
Sep 17 18:41:39.087532 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:39.087507    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/920d484e-62d1-4a98-aa61-b5669584dac5-kserve-provision-location\") pod \"tls-verification-test-kserve-85c4d7bc5b-pbqpl\" (UID: \"920d484e-62d1-4a98-aa61-b5669584dac5\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"
Sep 17 18:41:39.087647 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:39.087543    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920d484e-62d1-4a98-aa61-b5669584dac5-model-cache\") pod \"tls-verification-test-kserve-85c4d7bc5b-pbqpl\" (UID: \"920d484e-62d1-4a98-aa61-b5669584dac5\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"
Sep 17 18:41:39.087647 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:39.087574    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/920d484e-62d1-4a98-aa61-b5669584dac5-tls-certs\") pod \"tls-verification-test-kserve-85c4d7bc5b-pbqpl\" (UID: \"920d484e-62d1-4a98-aa61-b5669584dac5\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"
Sep 17 18:41:39.087647 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:39.087583    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/920d484e-62d1-4a98-aa61-b5669584dac5-home\") pod \"tls-verification-test-kserve-85c4d7bc5b-pbqpl\" (UID: \"920d484e-62d1-4a98-aa61-b5669584dac5\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"
Sep 17 18:41:39.087647 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:39.087604    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/920d484e-62d1-4a98-aa61-b5669584dac5-dshm\") pod \"tls-verification-test-kserve-85c4d7bc5b-pbqpl\" (UID: \"920d484e-62d1-4a98-aa61-b5669584dac5\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"
Sep 17 18:41:39.087908 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:39.087848    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920d484e-62d1-4a98-aa61-b5669584dac5-tmp-dir\") pod \"tls-verification-test-kserve-85c4d7bc5b-pbqpl\" (UID: \"920d484e-62d1-4a98-aa61-b5669584dac5\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"
Sep 17 18:41:39.087967 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:39.087924    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920d484e-62d1-4a98-aa61-b5669584dac5-model-cache\") pod \"tls-verification-test-kserve-85c4d7bc5b-pbqpl\" (UID: \"920d484e-62d1-4a98-aa61-b5669584dac5\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"
Sep 17 18:41:39.088060 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:39.088030    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/920d484e-62d1-4a98-aa61-b5669584dac5-cabundle-cert\") pod \"tls-verification-test-kserve-85c4d7bc5b-pbqpl\" (UID: \"920d484e-62d1-4a98-aa61-b5669584dac5\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"
Sep 17 18:41:39.088203 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:39.088119    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/920d484e-62d1-4a98-aa61-b5669584dac5-kserve-provision-location\") pod \"tls-verification-test-kserve-85c4d7bc5b-pbqpl\" (UID: \"920d484e-62d1-4a98-aa61-b5669584dac5\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"
Sep 17 18:41:39.089946 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:39.089928    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/920d484e-62d1-4a98-aa61-b5669584dac5-dshm\") pod \"tls-verification-test-kserve-85c4d7bc5b-pbqpl\" (UID: \"920d484e-62d1-4a98-aa61-b5669584dac5\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"
Sep 17 18:41:39.090436 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:39.090417    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/920d484e-62d1-4a98-aa61-b5669584dac5-tls-certs\") pod \"tls-verification-test-kserve-85c4d7bc5b-pbqpl\" (UID: \"920d484e-62d1-4a98-aa61-b5669584dac5\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"
Sep 17 18:41:39.095558 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:39.095527    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz7cb\" (UniqueName: \"kubernetes.io/projected/920d484e-62d1-4a98-aa61-b5669584dac5-kube-api-access-qz7cb\") pod \"tls-verification-test-kserve-85c4d7bc5b-pbqpl\" (UID: \"920d484e-62d1-4a98-aa61-b5669584dac5\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"
Sep 17 18:41:39.220315 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:39.220234    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"
Sep 17 18:41:39.501492 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:39.501467    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"]
Sep 17 18:41:39.504247 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:41:39.504219    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod920d484e_62d1_4a98_aa61_b5669584dac5.slice/crio-4cb48ac0cb660e80c73d85321551fc98af20c0ca334c6a689915c5567a3ac5ae WatchSource:0}: Error finding container 4cb48ac0cb660e80c73d85321551fc98af20c0ca334c6a689915c5567a3ac5ae: Status 404 returned error can't find the container with id 4cb48ac0cb660e80c73d85321551fc98af20c0ca334c6a689915c5567a3ac5ae
Sep 17 18:41:39.876346 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:39.876308    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl" event={"ID":"920d484e-62d1-4a98-aa61-b5669584dac5","Type":"ContainerStarted","Data":"38dbef3571cf5affd2f1f14c8c0a24b04d51c0d89f1527fec13faaaded88de5c"}
Sep 17 18:41:39.876546 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:39.876372    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl" event={"ID":"920d484e-62d1-4a98-aa61-b5669584dac5","Type":"ContainerStarted","Data":"4cb48ac0cb660e80c73d85321551fc98af20c0ca334c6a689915c5567a3ac5ae"}
Sep 17 18:41:41.886715 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:41.886684    2583 generic.go:356] "Generic (PLEG): container finished" podID="920d484e-62d1-4a98-aa61-b5669584dac5" containerID="38dbef3571cf5affd2f1f14c8c0a24b04d51c0d89f1527fec13faaaded88de5c" exitCode=0
Sep 17 18:41:41.887093 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:41.886747    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl" event={"ID":"920d484e-62d1-4a98-aa61-b5669584dac5","Type":"ContainerDied","Data":"38dbef3571cf5affd2f1f14c8c0a24b04d51c0d89f1527fec13faaaded88de5c"}
Sep 17 18:41:41.903689 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:41.903649    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8jf4l"]
Sep 17 18:41:41.911423 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:41.911400    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-8jf4l"]
Sep 17 18:41:42.893738 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:42.893699    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl" event={"ID":"920d484e-62d1-4a98-aa61-b5669584dac5","Type":"ContainerStarted","Data":"9c45b1abc1d5d672376c7d10ca4de72c3019f3a3041b4c7efd64cf7c210fd9a5"}
Sep 17 18:41:42.920739 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:42.920639    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl" podStartSLOduration=2.6325108569999998 podStartE2EDuration="4.92062018s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.288109323s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:41:42.919472733 +0000 UTC m=+4014.010779523" watchObservedRunningTime="2026-09-17 18:41:42.92062018 +0000 UTC m=+4014.011926946"
Sep 17 18:41:43.041112 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:43.041068    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="162810fc-3ab1-470a-985b-0df31ca2710a" path="/var/lib/kubelet/pods/162810fc-3ab1-470a-985b-0df31ca2710a/volumes"
Sep 17 18:41:43.221212 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:43.221125    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"
Sep 17 18:41:49.221048 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:49.221010    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"
Sep 17 18:41:49.222270 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:49.222240    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl" podUID="920d484e-62d1-4a98-aa61-b5669584dac5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 18:41:59.221503 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:41:59.221453    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl" podUID="920d484e-62d1-4a98-aa61-b5669584dac5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 18:42:09.221153 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:09.221056    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl" podUID="920d484e-62d1-4a98-aa61-b5669584dac5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 18:42:19.220983 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:19.220940    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl" podUID="920d484e-62d1-4a98-aa61-b5669584dac5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 18:42:21.073147 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:21.073111    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj"]
Sep 17 18:42:26.590683 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:26.590624    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj" podUID="48352ae8-12dd-4a0e-a30c-ea2e3e5639a3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:42:29.220773 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:29.220734    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl" podUID="920d484e-62d1-4a98-aa61-b5669584dac5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 18:42:36.101297 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:36.101225    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj" podUID="48352ae8-12dd-4a0e-a30c-ea2e3e5639a3" containerName="main" containerID="cri-o://8d462a27a9bb885874c6fa4c20323de555404814bad05562ab30b826f6db1f16" gracePeriod=45
Sep 17 18:42:36.590434 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:36.590347    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj" podUID="48352ae8-12dd-4a0e-a30c-ea2e3e5639a3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:42:37.602956 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:37.602915    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 18:42:39.220677 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:39.220627    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl" podUID="920d484e-62d1-4a98-aa61-b5669584dac5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 18:42:46.590822 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:46.590780    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj" podUID="48352ae8-12dd-4a0e-a30c-ea2e3e5639a3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:42:46.591310 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:46.590898    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj"
Sep 17 18:42:49.220860 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:49.220819    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl" podUID="920d484e-62d1-4a98-aa61-b5669584dac5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 18:42:51.445845 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:51.445808    2583 scope.go:117] "RemoveContainer" containerID="2f03621d6be0d5c0100a5f5357abf3481aa924ab757b27b10b31371921963430"
Sep 17 18:42:52.625981 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:52.625934    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="2762161a-5261-4ece-b75a-5a6d8e0c1878" containerName="main" containerID="cri-o://b59a35c467476e829dc0ac0c301022c8aa55dd008d1372c78ea0adb52b3959ea" gracePeriod=45
Sep 17 18:42:53.572327 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:53.572296    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:42:53.749966 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:53.749884    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2762161a-5261-4ece-b75a-5a6d8e0c1878-model-cache\") pod \"2762161a-5261-4ece-b75a-5a6d8e0c1878\" (UID: \"2762161a-5261-4ece-b75a-5a6d8e0c1878\") "
Sep 17 18:42:53.749966 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:53.749926    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppkfh\" (UniqueName: \"kubernetes.io/projected/2762161a-5261-4ece-b75a-5a6d8e0c1878-kube-api-access-ppkfh\") pod \"2762161a-5261-4ece-b75a-5a6d8e0c1878\" (UID: \"2762161a-5261-4ece-b75a-5a6d8e0c1878\") "
Sep 17 18:42:53.749966 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:53.749966    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2762161a-5261-4ece-b75a-5a6d8e0c1878-kserve-provision-location\") pod \"2762161a-5261-4ece-b75a-5a6d8e0c1878\" (UID: \"2762161a-5261-4ece-b75a-5a6d8e0c1878\") "
Sep 17 18:42:53.750546 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:53.750018    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2762161a-5261-4ece-b75a-5a6d8e0c1878-tls-certs\") pod \"2762161a-5261-4ece-b75a-5a6d8e0c1878\" (UID: \"2762161a-5261-4ece-b75a-5a6d8e0c1878\") "
Sep 17 18:42:53.750546 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:53.750069    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2762161a-5261-4ece-b75a-5a6d8e0c1878-tmp-dir\") pod \"2762161a-5261-4ece-b75a-5a6d8e0c1878\" (UID: \"2762161a-5261-4ece-b75a-5a6d8e0c1878\") "
Sep 17 18:42:53.750546 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:53.750136    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2762161a-5261-4ece-b75a-5a6d8e0c1878-cabundle-cert\") pod \"2762161a-5261-4ece-b75a-5a6d8e0c1878\" (UID: \"2762161a-5261-4ece-b75a-5a6d8e0c1878\") "
Sep 17 18:42:53.750546 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:53.750164    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2762161a-5261-4ece-b75a-5a6d8e0c1878-dshm\") pod \"2762161a-5261-4ece-b75a-5a6d8e0c1878\" (UID: \"2762161a-5261-4ece-b75a-5a6d8e0c1878\") "
Sep 17 18:42:53.750546 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:53.750203    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2762161a-5261-4ece-b75a-5a6d8e0c1878-home\") pod \"2762161a-5261-4ece-b75a-5a6d8e0c1878\" (UID: \"2762161a-5261-4ece-b75a-5a6d8e0c1878\") "
Sep 17 18:42:53.750546 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:53.750199    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2762161a-5261-4ece-b75a-5a6d8e0c1878-model-cache" (OuterVolumeSpecName: "model-cache") pod "2762161a-5261-4ece-b75a-5a6d8e0c1878" (UID: "2762161a-5261-4ece-b75a-5a6d8e0c1878"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:42:53.750546 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:53.750491    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2762161a-5261-4ece-b75a-5a6d8e0c1878-model-cache\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:42:53.750546 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:53.750524    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2762161a-5261-4ece-b75a-5a6d8e0c1878-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2762161a-5261-4ece-b75a-5a6d8e0c1878" (UID: "2762161a-5261-4ece-b75a-5a6d8e0c1878"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:42:53.751022 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:53.750984    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2762161a-5261-4ece-b75a-5a6d8e0c1878-home" (OuterVolumeSpecName: "home") pod "2762161a-5261-4ece-b75a-5a6d8e0c1878" (UID: "2762161a-5261-4ece-b75a-5a6d8e0c1878"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:42:53.752786 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:53.752759    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2762161a-5261-4ece-b75a-5a6d8e0c1878-dshm" (OuterVolumeSpecName: "dshm") pod "2762161a-5261-4ece-b75a-5a6d8e0c1878" (UID: "2762161a-5261-4ece-b75a-5a6d8e0c1878"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:42:53.752891 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:53.752806    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2762161a-5261-4ece-b75a-5a6d8e0c1878-kube-api-access-ppkfh" (OuterVolumeSpecName: "kube-api-access-ppkfh") pod "2762161a-5261-4ece-b75a-5a6d8e0c1878" (UID: "2762161a-5261-4ece-b75a-5a6d8e0c1878"). InnerVolumeSpecName "kube-api-access-ppkfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:42:53.752891 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:53.752873    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2762161a-5261-4ece-b75a-5a6d8e0c1878-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2762161a-5261-4ece-b75a-5a6d8e0c1878" (UID: "2762161a-5261-4ece-b75a-5a6d8e0c1878"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:42:53.768864 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:53.768829    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2762161a-5261-4ece-b75a-5a6d8e0c1878-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2762161a-5261-4ece-b75a-5a6d8e0c1878" (UID: "2762161a-5261-4ece-b75a-5a6d8e0c1878"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:42:53.774846 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:53.774822    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2762161a-5261-4ece-b75a-5a6d8e0c1878-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2762161a-5261-4ece-b75a-5a6d8e0c1878" (UID: "2762161a-5261-4ece-b75a-5a6d8e0c1878"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:42:53.851817 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:53.851781    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2762161a-5261-4ece-b75a-5a6d8e0c1878-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:42:53.851817 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:53.851810    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2762161a-5261-4ece-b75a-5a6d8e0c1878-tmp-dir\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:42:53.851817 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:53.851819    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2762161a-5261-4ece-b75a-5a6d8e0c1878-cabundle-cert\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:42:53.851817 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:53.851827    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2762161a-5261-4ece-b75a-5a6d8e0c1878-dshm\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:42:53.852190 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:53.851837    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2762161a-5261-4ece-b75a-5a6d8e0c1878-home\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:42:53.852190 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:53.851845    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppkfh\" (UniqueName: \"kubernetes.io/projected/2762161a-5261-4ece-b75a-5a6d8e0c1878-kube-api-access-ppkfh\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:42:53.852190 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:53.851853    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2762161a-5261-4ece-b75a-5a6d8e0c1878-kserve-provision-location\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:42:54.239304 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:54.239267    2583 generic.go:356] "Generic (PLEG): container finished" podID="2762161a-5261-4ece-b75a-5a6d8e0c1878" containerID="b59a35c467476e829dc0ac0c301022c8aa55dd008d1372c78ea0adb52b3959ea" exitCode=0
Sep 17 18:42:54.239519 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:54.239347    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 17 18:42:54.239519 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:54.239362    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"2762161a-5261-4ece-b75a-5a6d8e0c1878","Type":"ContainerDied","Data":"b59a35c467476e829dc0ac0c301022c8aa55dd008d1372c78ea0adb52b3959ea"}
Sep 17 18:42:54.239519 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:54.239435    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"2762161a-5261-4ece-b75a-5a6d8e0c1878","Type":"ContainerDied","Data":"e6789ea1faa57567bb84d111ce369f02ae9a28ea3cce523e4bee200ccf0d0fd8"}
Sep 17 18:42:54.239519 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:54.239467    2583 scope.go:117] "RemoveContainer" containerID="b59a35c467476e829dc0ac0c301022c8aa55dd008d1372c78ea0adb52b3959ea"
Sep 17 18:42:54.248031 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:54.248015    2583 scope.go:117] "RemoveContainer" containerID="f3f329dde7b0ed200a8edde0feceacf61b1107860e09f25412c7ba879d67b5b4"
Sep 17 18:42:54.255819 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:54.255796    2583 scope.go:117] "RemoveContainer" containerID="b59a35c467476e829dc0ac0c301022c8aa55dd008d1372c78ea0adb52b3959ea"
Sep 17 18:42:54.256094 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:42:54.256076    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b59a35c467476e829dc0ac0c301022c8aa55dd008d1372c78ea0adb52b3959ea\": container with ID starting with b59a35c467476e829dc0ac0c301022c8aa55dd008d1372c78ea0adb52b3959ea not found: ID does not exist" containerID="b59a35c467476e829dc0ac0c301022c8aa55dd008d1372c78ea0adb52b3959ea"
Sep 17 18:42:54.256177 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:54.256099    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b59a35c467476e829dc0ac0c301022c8aa55dd008d1372c78ea0adb52b3959ea"} err="failed to get container status \"b59a35c467476e829dc0ac0c301022c8aa55dd008d1372c78ea0adb52b3959ea\": rpc error: code = NotFound desc = could not find container \"b59a35c467476e829dc0ac0c301022c8aa55dd008d1372c78ea0adb52b3959ea\": container with ID starting with b59a35c467476e829dc0ac0c301022c8aa55dd008d1372c78ea0adb52b3959ea not found: ID does not exist"
Sep 17 18:42:54.256177 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:54.256115    2583 scope.go:117] "RemoveContainer" containerID="f3f329dde7b0ed200a8edde0feceacf61b1107860e09f25412c7ba879d67b5b4"
Sep 17 18:42:54.256374 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:42:54.256354    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3f329dde7b0ed200a8edde0feceacf61b1107860e09f25412c7ba879d67b5b4\": container with ID starting with f3f329dde7b0ed200a8edde0feceacf61b1107860e09f25412c7ba879d67b5b4 not found: ID does not exist" containerID="f3f329dde7b0ed200a8edde0feceacf61b1107860e09f25412c7ba879d67b5b4"
Sep 17 18:42:54.256427 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:54.256380    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3f329dde7b0ed200a8edde0feceacf61b1107860e09f25412c7ba879d67b5b4"} err="failed to get container status \"f3f329dde7b0ed200a8edde0feceacf61b1107860e09f25412c7ba879d67b5b4\": rpc error: code = NotFound desc = could not find container \"f3f329dde7b0ed200a8edde0feceacf61b1107860e09f25412c7ba879d67b5b4\": container with ID starting with f3f329dde7b0ed200a8edde0feceacf61b1107860e09f25412c7ba879d67b5b4 not found: ID does not exist"
Sep 17 18:42:54.267477 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:54.267456    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 18:42:54.271591 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:54.271571    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 17 18:42:55.041161 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:55.041128    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2762161a-5261-4ece-b75a-5a6d8e0c1878" path="/var/lib/kubelet/pods/2762161a-5261-4ece-b75a-5a6d8e0c1878/volumes"
Sep 17 18:42:56.590572 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:56.590536    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj" podUID="48352ae8-12dd-4a0e-a30c-ea2e3e5639a3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:42:59.221490 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:42:59.221456    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl" podUID="920d484e-62d1-4a98-aa61-b5669584dac5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 18:43:06.302598 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:06.302557    2583 generic.go:356] "Generic (PLEG): container finished" podID="48352ae8-12dd-4a0e-a30c-ea2e3e5639a3" containerID="8d462a27a9bb885874c6fa4c20323de555404814bad05562ab30b826f6db1f16" exitCode=0
Sep 17 18:43:06.302984 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:06.302611    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj" event={"ID":"48352ae8-12dd-4a0e-a30c-ea2e3e5639a3","Type":"ContainerDied","Data":"8d462a27a9bb885874c6fa4c20323de555404814bad05562ab30b826f6db1f16"}
Sep 17 18:43:06.342606 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:06.342584    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj"
Sep 17 18:43:06.358874 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:06.358816    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpg97\" (UniqueName: \"kubernetes.io/projected/48352ae8-12dd-4a0e-a30c-ea2e3e5639a3-kube-api-access-xpg97\") pod \"48352ae8-12dd-4a0e-a30c-ea2e3e5639a3\" (UID: \"48352ae8-12dd-4a0e-a30c-ea2e3e5639a3\") "
Sep 17 18:43:06.358989 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:06.358887    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48352ae8-12dd-4a0e-a30c-ea2e3e5639a3-tls-certs\") pod \"48352ae8-12dd-4a0e-a30c-ea2e3e5639a3\" (UID: \"48352ae8-12dd-4a0e-a30c-ea2e3e5639a3\") "
Sep 17 18:43:06.361358 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:06.361331    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48352ae8-12dd-4a0e-a30c-ea2e3e5639a3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "48352ae8-12dd-4a0e-a30c-ea2e3e5639a3" (UID: "48352ae8-12dd-4a0e-a30c-ea2e3e5639a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:43:06.362160 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:06.362134    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48352ae8-12dd-4a0e-a30c-ea2e3e5639a3-kube-api-access-xpg97" (OuterVolumeSpecName: "kube-api-access-xpg97") pod "48352ae8-12dd-4a0e-a30c-ea2e3e5639a3" (UID: "48352ae8-12dd-4a0e-a30c-ea2e3e5639a3"). InnerVolumeSpecName "kube-api-access-xpg97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:43:06.459622 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:06.459571    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48352ae8-12dd-4a0e-a30c-ea2e3e5639a3-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:43:06.459622 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:06.459616    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpg97\" (UniqueName: \"kubernetes.io/projected/48352ae8-12dd-4a0e-a30c-ea2e3e5639a3-kube-api-access-xpg97\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:43:07.307895 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:07.307811    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj" event={"ID":"48352ae8-12dd-4a0e-a30c-ea2e3e5639a3","Type":"ContainerDied","Data":"fc6855578b125b28f5da684f846734d481c68de6e453d5ffdaadcfecb895ec07"}
Sep 17 18:43:07.307895 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:07.307872    2583 scope.go:117] "RemoveContainer" containerID="8d462a27a9bb885874c6fa4c20323de555404814bad05562ab30b826f6db1f16"
Sep 17 18:43:07.308386 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:07.307823    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj"
Sep 17 18:43:07.325290 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:07.325266    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj"]
Sep 17 18:43:07.333509 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:07.333486    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schem8mjj"]
Sep 17 18:43:09.041037 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:09.040991    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48352ae8-12dd-4a0e-a30c-ea2e3e5639a3" path="/var/lib/kubelet/pods/48352ae8-12dd-4a0e-a30c-ea2e3e5639a3/volumes"
Sep 17 18:43:09.221390 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:09.221352    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl" podUID="920d484e-62d1-4a98-aa61-b5669584dac5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 18:43:19.221026 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:19.220988    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl" podUID="920d484e-62d1-4a98-aa61-b5669584dac5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 18:43:20.861551 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:20.861519    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k"]
Sep 17 18:43:20.861989 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:20.861860    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48352ae8-12dd-4a0e-a30c-ea2e3e5639a3" containerName="main"
Sep 17 18:43:20.861989 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:20.861876    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="48352ae8-12dd-4a0e-a30c-ea2e3e5639a3" containerName="main"
Sep 17 18:43:20.861989 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:20.861889    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2762161a-5261-4ece-b75a-5a6d8e0c1878" containerName="storage-initializer"
Sep 17 18:43:20.861989 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:20.861898    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2762161a-5261-4ece-b75a-5a6d8e0c1878" containerName="storage-initializer"
Sep 17 18:43:20.861989 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:20.861934    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2762161a-5261-4ece-b75a-5a6d8e0c1878" containerName="main"
Sep 17 18:43:20.861989 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:20.861943    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2762161a-5261-4ece-b75a-5a6d8e0c1878" containerName="main"
Sep 17 18:43:20.862192 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:20.862024    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="48352ae8-12dd-4a0e-a30c-ea2e3e5639a3" containerName="main"
Sep 17 18:43:20.862192 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:20.862041    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2762161a-5261-4ece-b75a-5a6d8e0c1878" containerName="main"
Sep 17 18:43:20.867719 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:20.867700    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k"
Sep 17 18:43:20.871438 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:20.871412    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-5rqzv\""
Sep 17 18:43:20.871561 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:20.871433    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 17 18:43:20.871561 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:20.871450    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 17 18:43:20.871561 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:20.871412    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 17 18:43:20.871561 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:20.871493    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-kkpfg\""
Sep 17 18:43:20.879365 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:20.879343    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k"]
Sep 17 18:43:20.986387 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:20.986352    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8t7c\" (UniqueName: \"kubernetes.io/projected/32e238d3-4b57-4bd1-bc32-c96b68cbfed9-kube-api-access-w8t7c\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k\" (UID: \"32e238d3-4b57-4bd1-bc32-c96b68cbfed9\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k"
Sep 17 18:43:20.986616 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:20.986458    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32e238d3-4b57-4bd1-bc32-c96b68cbfed9-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k\" (UID: \"32e238d3-4b57-4bd1-bc32-c96b68cbfed9\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k"
Sep 17 18:43:21.087039 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:21.086995    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32e238d3-4b57-4bd1-bc32-c96b68cbfed9-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k\" (UID: \"32e238d3-4b57-4bd1-bc32-c96b68cbfed9\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k"
Sep 17 18:43:21.087242 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:21.087056    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8t7c\" (UniqueName: \"kubernetes.io/projected/32e238d3-4b57-4bd1-bc32-c96b68cbfed9-kube-api-access-w8t7c\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k\" (UID: \"32e238d3-4b57-4bd1-bc32-c96b68cbfed9\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k"
Sep 17 18:43:21.089566 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:21.089539    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32e238d3-4b57-4bd1-bc32-c96b68cbfed9-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k\" (UID: \"32e238d3-4b57-4bd1-bc32-c96b68cbfed9\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k"
Sep 17 18:43:21.095161 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:21.095134    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8t7c\" (UniqueName: \"kubernetes.io/projected/32e238d3-4b57-4bd1-bc32-c96b68cbfed9-kube-api-access-w8t7c\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k\" (UID: \"32e238d3-4b57-4bd1-bc32-c96b68cbfed9\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k"
Sep 17 18:43:21.178844 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:21.178744    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k"
Sep 17 18:43:21.455234 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:21.455141    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k"]
Sep 17 18:43:21.459553 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:43:21.459523    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e238d3_4b57_4bd1_bc32_c96b68cbfed9.slice/crio-48c4407d3162a53abbf2e4861f99456de2baee96a329e22b22c54e38a3299820 WatchSource:0}: Error finding container 48c4407d3162a53abbf2e4861f99456de2baee96a329e22b22c54e38a3299820: Status 404 returned error can't find the container with id 48c4407d3162a53abbf2e4861f99456de2baee96a329e22b22c54e38a3299820
Sep 17 18:43:22.380062 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:22.380030    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k" event={"ID":"32e238d3-4b57-4bd1-bc32-c96b68cbfed9","Type":"ContainerStarted","Data":"1ccb2ac3e03d8a3b5b120a5b0e07131e74bb1d0e57e3a80cede7ddf0fcc7c160"}
Sep 17 18:43:22.380503 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:22.380080    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k" event={"ID":"32e238d3-4b57-4bd1-bc32-c96b68cbfed9","Type":"ContainerStarted","Data":"48c4407d3162a53abbf2e4861f99456de2baee96a329e22b22c54e38a3299820"}
Sep 17 18:43:22.380503 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:22.380170    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k"
Sep 17 18:43:22.403120 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:22.403069    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k" podStartSLOduration=2.403052697 podStartE2EDuration="2.403052697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:43:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:43:22.40128927 +0000 UTC m=+4113.492596075" watchObservedRunningTime="2026-09-17 18:43:22.403052697 +0000 UTC m=+4113.494359462"
Sep 17 18:43:29.221407 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:29.221369    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl" podUID="920d484e-62d1-4a98-aa61-b5669584dac5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 18:43:39.231150 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:39.231120    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"
Sep 17 18:43:39.239109 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:39.239084    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"
Sep 17 18:43:44.083800 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:44.083766    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"]
Sep 17 18:43:53.386441 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:53.386408    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k"
Sep 17 18:43:54.930214 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:54.930181    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k"]
Sep 17 18:43:59.106459 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:59.106396    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl" podUID="920d484e-62d1-4a98-aa61-b5669584dac5" containerName="main" containerID="cri-o://9c45b1abc1d5d672376c7d10ca4de72c3019f3a3041b4c7efd64cf7c210fd9a5" gracePeriod=45
Sep 17 18:43:59.232016 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:59.231976    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl" podUID="920d484e-62d1-4a98-aa61-b5669584dac5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 17 18:43:59.842378 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:59.842350    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"
Sep 17 18:43:59.901690 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:59.901618    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/920d484e-62d1-4a98-aa61-b5669584dac5-cabundle-cert\") pod \"920d484e-62d1-4a98-aa61-b5669584dac5\" (UID: \"920d484e-62d1-4a98-aa61-b5669584dac5\") "
Sep 17 18:43:59.901690 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:59.901650    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920d484e-62d1-4a98-aa61-b5669584dac5-tmp-dir\") pod \"920d484e-62d1-4a98-aa61-b5669584dac5\" (UID: \"920d484e-62d1-4a98-aa61-b5669584dac5\") "
Sep 17 18:43:59.901897 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:59.901697    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/920d484e-62d1-4a98-aa61-b5669584dac5-tls-certs\") pod \"920d484e-62d1-4a98-aa61-b5669584dac5\" (UID: \"920d484e-62d1-4a98-aa61-b5669584dac5\") "
Sep 17 18:43:59.901897 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:59.901725    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/920d484e-62d1-4a98-aa61-b5669584dac5-kserve-provision-location\") pod \"920d484e-62d1-4a98-aa61-b5669584dac5\" (UID: \"920d484e-62d1-4a98-aa61-b5669584dac5\") "
Sep 17 18:43:59.901897 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:59.901742    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/920d484e-62d1-4a98-aa61-b5669584dac5-home\") pod \"920d484e-62d1-4a98-aa61-b5669584dac5\" (UID: \"920d484e-62d1-4a98-aa61-b5669584dac5\") "
Sep 17 18:43:59.901897 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:59.901765    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920d484e-62d1-4a98-aa61-b5669584dac5-model-cache\") pod \"920d484e-62d1-4a98-aa61-b5669584dac5\" (UID: \"920d484e-62d1-4a98-aa61-b5669584dac5\") "
Sep 17 18:43:59.901897 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:59.901865    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/920d484e-62d1-4a98-aa61-b5669584dac5-dshm\") pod \"920d484e-62d1-4a98-aa61-b5669584dac5\" (UID: \"920d484e-62d1-4a98-aa61-b5669584dac5\") "
Sep 17 18:43:59.902162 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:59.901935    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qz7cb\" (UniqueName: \"kubernetes.io/projected/920d484e-62d1-4a98-aa61-b5669584dac5-kube-api-access-qz7cb\") pod \"920d484e-62d1-4a98-aa61-b5669584dac5\" (UID: \"920d484e-62d1-4a98-aa61-b5669584dac5\") "
Sep 17 18:43:59.902162 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:59.902024    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920d484e-62d1-4a98-aa61-b5669584dac5-model-cache" (OuterVolumeSpecName: "model-cache") pod "920d484e-62d1-4a98-aa61-b5669584dac5" (UID: "920d484e-62d1-4a98-aa61-b5669584dac5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:43:59.902270 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:59.902190    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/920d484e-62d1-4a98-aa61-b5669584dac5-model-cache\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:43:59.902361 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:59.902336    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/920d484e-62d1-4a98-aa61-b5669584dac5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "920d484e-62d1-4a98-aa61-b5669584dac5" (UID: "920d484e-62d1-4a98-aa61-b5669584dac5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:43:59.902508 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:59.902478    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920d484e-62d1-4a98-aa61-b5669584dac5-home" (OuterVolumeSpecName: "home") pod "920d484e-62d1-4a98-aa61-b5669584dac5" (UID: "920d484e-62d1-4a98-aa61-b5669584dac5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:43:59.904222 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:59.904188    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/920d484e-62d1-4a98-aa61-b5669584dac5-kube-api-access-qz7cb" (OuterVolumeSpecName: "kube-api-access-qz7cb") pod "920d484e-62d1-4a98-aa61-b5669584dac5" (UID: "920d484e-62d1-4a98-aa61-b5669584dac5"). InnerVolumeSpecName "kube-api-access-qz7cb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:43:59.904350 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:59.904231    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920d484e-62d1-4a98-aa61-b5669584dac5-dshm" (OuterVolumeSpecName: "dshm") pod "920d484e-62d1-4a98-aa61-b5669584dac5" (UID: "920d484e-62d1-4a98-aa61-b5669584dac5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:43:59.904350 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:59.904309    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/920d484e-62d1-4a98-aa61-b5669584dac5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "920d484e-62d1-4a98-aa61-b5669584dac5" (UID: "920d484e-62d1-4a98-aa61-b5669584dac5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:43:59.915496 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:59.915454    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920d484e-62d1-4a98-aa61-b5669584dac5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "920d484e-62d1-4a98-aa61-b5669584dac5" (UID: "920d484e-62d1-4a98-aa61-b5669584dac5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:43:59.923928 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:43:59.923906    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/920d484e-62d1-4a98-aa61-b5669584dac5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "920d484e-62d1-4a98-aa61-b5669584dac5" (UID: "920d484e-62d1-4a98-aa61-b5669584dac5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:44:00.003142 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:00.003118    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz7cb\" (UniqueName: \"kubernetes.io/projected/920d484e-62d1-4a98-aa61-b5669584dac5-kube-api-access-qz7cb\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:44:00.003142 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:00.003139    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/920d484e-62d1-4a98-aa61-b5669584dac5-cabundle-cert\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:44:00.003277 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:00.003149    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/920d484e-62d1-4a98-aa61-b5669584dac5-tmp-dir\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:44:00.003277 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:00.003157    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/920d484e-62d1-4a98-aa61-b5669584dac5-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:44:00.003277 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:00.003165    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/920d484e-62d1-4a98-aa61-b5669584dac5-kserve-provision-location\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:44:00.003277 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:00.003175    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/920d484e-62d1-4a98-aa61-b5669584dac5-home\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:44:00.003277 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:00.003188    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/920d484e-62d1-4a98-aa61-b5669584dac5-dshm\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:44:00.551329 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:00.551296    2583 generic.go:356] "Generic (PLEG): container finished" podID="920d484e-62d1-4a98-aa61-b5669584dac5" containerID="9c45b1abc1d5d672376c7d10ca4de72c3019f3a3041b4c7efd64cf7c210fd9a5" exitCode=0
Sep 17 18:44:00.551796 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:00.551353    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"
Sep 17 18:44:00.551796 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:00.551393    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl" event={"ID":"920d484e-62d1-4a98-aa61-b5669584dac5","Type":"ContainerDied","Data":"9c45b1abc1d5d672376c7d10ca4de72c3019f3a3041b4c7efd64cf7c210fd9a5"}
Sep 17 18:44:00.551796 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:00.551450    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl" event={"ID":"920d484e-62d1-4a98-aa61-b5669584dac5","Type":"ContainerDied","Data":"4cb48ac0cb660e80c73d85321551fc98af20c0ca334c6a689915c5567a3ac5ae"}
Sep 17 18:44:00.551796 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:00.551474    2583 scope.go:117] "RemoveContainer" containerID="9c45b1abc1d5d672376c7d10ca4de72c3019f3a3041b4c7efd64cf7c210fd9a5"
Sep 17 18:44:00.560180 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:00.560165    2583 scope.go:117] "RemoveContainer" containerID="38dbef3571cf5affd2f1f14c8c0a24b04d51c0d89f1527fec13faaaded88de5c"
Sep 17 18:44:00.567429 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:00.567412    2583 scope.go:117] "RemoveContainer" containerID="9c45b1abc1d5d672376c7d10ca4de72c3019f3a3041b4c7efd64cf7c210fd9a5"
Sep 17 18:44:00.567701 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:44:00.567680    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c45b1abc1d5d672376c7d10ca4de72c3019f3a3041b4c7efd64cf7c210fd9a5\": container with ID starting with 9c45b1abc1d5d672376c7d10ca4de72c3019f3a3041b4c7efd64cf7c210fd9a5 not found: ID does not exist" containerID="9c45b1abc1d5d672376c7d10ca4de72c3019f3a3041b4c7efd64cf7c210fd9a5"
Sep 17 18:44:00.567787 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:00.567712    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c45b1abc1d5d672376c7d10ca4de72c3019f3a3041b4c7efd64cf7c210fd9a5"} err="failed to get container status \"9c45b1abc1d5d672376c7d10ca4de72c3019f3a3041b4c7efd64cf7c210fd9a5\": rpc error: code = NotFound desc = could not find container \"9c45b1abc1d5d672376c7d10ca4de72c3019f3a3041b4c7efd64cf7c210fd9a5\": container with ID starting with 9c45b1abc1d5d672376c7d10ca4de72c3019f3a3041b4c7efd64cf7c210fd9a5 not found: ID does not exist"
Sep 17 18:44:00.567787 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:00.567735    2583 scope.go:117] "RemoveContainer" containerID="38dbef3571cf5affd2f1f14c8c0a24b04d51c0d89f1527fec13faaaded88de5c"
Sep 17 18:44:00.567989 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:44:00.567970    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38dbef3571cf5affd2f1f14c8c0a24b04d51c0d89f1527fec13faaaded88de5c\": container with ID starting with 38dbef3571cf5affd2f1f14c8c0a24b04d51c0d89f1527fec13faaaded88de5c not found: ID does not exist" containerID="38dbef3571cf5affd2f1f14c8c0a24b04d51c0d89f1527fec13faaaded88de5c"
Sep 17 18:44:00.568034 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:00.567995    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38dbef3571cf5affd2f1f14c8c0a24b04d51c0d89f1527fec13faaaded88de5c"} err="failed to get container status \"38dbef3571cf5affd2f1f14c8c0a24b04d51c0d89f1527fec13faaaded88de5c\": rpc error: code = NotFound desc = could not find container \"38dbef3571cf5affd2f1f14c8c0a24b04d51c0d89f1527fec13faaaded88de5c\": container with ID starting with 38dbef3571cf5affd2f1f14c8c0a24b04d51c0d89f1527fec13faaaded88de5c not found: ID does not exist"
Sep 17 18:44:00.574221 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:00.574201    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"]
Sep 17 18:44:00.577926 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:00.577906    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-85c4d7bc5b-pbqpl"]
Sep 17 18:44:01.045477 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:01.045439    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="920d484e-62d1-4a98-aa61-b5669584dac5" path="/var/lib/kubelet/pods/920d484e-62d1-4a98-aa61-b5669584dac5/volumes"
Sep 17 18:44:03.386589 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:03.386547    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k" podUID="32e238d3-4b57-4bd1-bc32-c96b68cbfed9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:44:09.950308 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:09.950241    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k" podUID="32e238d3-4b57-4bd1-bc32-c96b68cbfed9" containerName="main" containerID="cri-o://1ccb2ac3e03d8a3b5b120a5b0e07131e74bb1d0e57e3a80cede7ddf0fcc7c160" gracePeriod=45
Sep 17 18:44:13.386794 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:13.386755    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k" podUID="32e238d3-4b57-4bd1-bc32-c96b68cbfed9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:44:23.386655 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:23.386615    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k" podUID="32e238d3-4b57-4bd1-bc32-c96b68cbfed9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:44:23.387117 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:23.386748    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k"
Sep 17 18:44:33.386576 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:33.386533    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k" podUID="32e238d3-4b57-4bd1-bc32-c96b68cbfed9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:44:40.190021 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:40.189995    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k"
Sep 17 18:44:40.316395 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:40.316319    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8t7c\" (UniqueName: \"kubernetes.io/projected/32e238d3-4b57-4bd1-bc32-c96b68cbfed9-kube-api-access-w8t7c\") pod \"32e238d3-4b57-4bd1-bc32-c96b68cbfed9\" (UID: \"32e238d3-4b57-4bd1-bc32-c96b68cbfed9\") "
Sep 17 18:44:40.316395 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:40.316379    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32e238d3-4b57-4bd1-bc32-c96b68cbfed9-tls-certs\") pod \"32e238d3-4b57-4bd1-bc32-c96b68cbfed9\" (UID: \"32e238d3-4b57-4bd1-bc32-c96b68cbfed9\") "
Sep 17 18:44:40.318432 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:40.318400    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32e238d3-4b57-4bd1-bc32-c96b68cbfed9-kube-api-access-w8t7c" (OuterVolumeSpecName: "kube-api-access-w8t7c") pod "32e238d3-4b57-4bd1-bc32-c96b68cbfed9" (UID: "32e238d3-4b57-4bd1-bc32-c96b68cbfed9"). InnerVolumeSpecName "kube-api-access-w8t7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:44:40.318559 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:40.318501    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32e238d3-4b57-4bd1-bc32-c96b68cbfed9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "32e238d3-4b57-4bd1-bc32-c96b68cbfed9" (UID: "32e238d3-4b57-4bd1-bc32-c96b68cbfed9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:44:40.417547 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:40.417517    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32e238d3-4b57-4bd1-bc32-c96b68cbfed9-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:44:40.417547 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:40.417544    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8t7c\" (UniqueName: \"kubernetes.io/projected/32e238d3-4b57-4bd1-bc32-c96b68cbfed9-kube-api-access-w8t7c\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:44:40.723515 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:40.723477    2583 generic.go:356] "Generic (PLEG): container finished" podID="32e238d3-4b57-4bd1-bc32-c96b68cbfed9" containerID="1ccb2ac3e03d8a3b5b120a5b0e07131e74bb1d0e57e3a80cede7ddf0fcc7c160" exitCode=0
Sep 17 18:44:40.723702 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:40.723527    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k"
Sep 17 18:44:40.723702 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:40.723556    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k" event={"ID":"32e238d3-4b57-4bd1-bc32-c96b68cbfed9","Type":"ContainerDied","Data":"1ccb2ac3e03d8a3b5b120a5b0e07131e74bb1d0e57e3a80cede7ddf0fcc7c160"}
Sep 17 18:44:40.723702 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:40.723611    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k" event={"ID":"32e238d3-4b57-4bd1-bc32-c96b68cbfed9","Type":"ContainerDied","Data":"48c4407d3162a53abbf2e4861f99456de2baee96a329e22b22c54e38a3299820"}
Sep 17 18:44:40.723702 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:40.723636    2583 scope.go:117] "RemoveContainer" containerID="1ccb2ac3e03d8a3b5b120a5b0e07131e74bb1d0e57e3a80cede7ddf0fcc7c160"
Sep 17 18:44:40.732591 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:40.732573    2583 scope.go:117] "RemoveContainer" containerID="1ccb2ac3e03d8a3b5b120a5b0e07131e74bb1d0e57e3a80cede7ddf0fcc7c160"
Sep 17 18:44:40.732853 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:44:40.732834    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ccb2ac3e03d8a3b5b120a5b0e07131e74bb1d0e57e3a80cede7ddf0fcc7c160\": container with ID starting with 1ccb2ac3e03d8a3b5b120a5b0e07131e74bb1d0e57e3a80cede7ddf0fcc7c160 not found: ID does not exist" containerID="1ccb2ac3e03d8a3b5b120a5b0e07131e74bb1d0e57e3a80cede7ddf0fcc7c160"
Sep 17 18:44:40.732912 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:40.732861    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ccb2ac3e03d8a3b5b120a5b0e07131e74bb1d0e57e3a80cede7ddf0fcc7c160"} err="failed to get container status \"1ccb2ac3e03d8a3b5b120a5b0e07131e74bb1d0e57e3a80cede7ddf0fcc7c160\": rpc error: code = NotFound desc = could not find container \"1ccb2ac3e03d8a3b5b120a5b0e07131e74bb1d0e57e3a80cede7ddf0fcc7c160\": container with ID starting with 1ccb2ac3e03d8a3b5b120a5b0e07131e74bb1d0e57e3a80cede7ddf0fcc7c160 not found: ID does not exist"
Sep 17 18:44:40.743713 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:40.743688    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k"]
Sep 17 18:44:40.747337 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:40.747308    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbqhl8k"]
Sep 17 18:44:41.040218 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:41.040140    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32e238d3-4b57-4bd1-bc32-c96b68cbfed9" path="/var/lib/kubelet/pods/32e238d3-4b57-4bd1-bc32-c96b68cbfed9/volumes"
Sep 17 18:44:45.427935 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.427897    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"]
Sep 17 18:44:45.428387 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.428163    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32e238d3-4b57-4bd1-bc32-c96b68cbfed9" containerName="main"
Sep 17 18:44:45.428387 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.428174    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="32e238d3-4b57-4bd1-bc32-c96b68cbfed9" containerName="main"
Sep 17 18:44:45.428387 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.428188    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="920d484e-62d1-4a98-aa61-b5669584dac5" containerName="main"
Sep 17 18:44:45.428387 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.428193    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="920d484e-62d1-4a98-aa61-b5669584dac5" containerName="main"
Sep 17 18:44:45.428387 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.428202    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="920d484e-62d1-4a98-aa61-b5669584dac5" containerName="storage-initializer"
Sep 17 18:44:45.428387 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.428208    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="920d484e-62d1-4a98-aa61-b5669584dac5" containerName="storage-initializer"
Sep 17 18:44:45.428387 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.428263    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="920d484e-62d1-4a98-aa61-b5669584dac5" containerName="main"
Sep 17 18:44:45.428387 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.428270    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="32e238d3-4b57-4bd1-bc32-c96b68cbfed9" containerName="main"
Sep 17 18:44:45.432431 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.432414    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"
Sep 17 18:44:45.434966 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.434940    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 17 18:44:45.435113 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.435056    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-65nmv\""
Sep 17 18:44:45.435395 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.435372    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 17 18:44:45.435504 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.435452    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 17 18:44:45.435930 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.435910    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 17 18:44:45.436106 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.436092    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:44:45.445249 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.445223    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"]
Sep 17 18:44:45.551820 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.551787    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a5121e0-0201-4024-bc90-6881d07bd62f-dshm\") pod \"prestop-hook-test-kserve-5f6b9b8bc6-9ksjx\" (UID: \"6a5121e0-0201-4024-bc90-6881d07bd62f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"
Sep 17 18:44:45.551989 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.551826    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a5121e0-0201-4024-bc90-6881d07bd62f-tls-certs\") pod \"prestop-hook-test-kserve-5f6b9b8bc6-9ksjx\" (UID: \"6a5121e0-0201-4024-bc90-6881d07bd62f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"
Sep 17 18:44:45.551989 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.551851    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a5121e0-0201-4024-bc90-6881d07bd62f-tmp-dir\") pod \"prestop-hook-test-kserve-5f6b9b8bc6-9ksjx\" (UID: \"6a5121e0-0201-4024-bc90-6881d07bd62f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"
Sep 17 18:44:45.551989 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.551879    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a5121e0-0201-4024-bc90-6881d07bd62f-model-cache\") pod \"prestop-hook-test-kserve-5f6b9b8bc6-9ksjx\" (UID: \"6a5121e0-0201-4024-bc90-6881d07bd62f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"
Sep 17 18:44:45.551989 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.551900    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a5121e0-0201-4024-bc90-6881d07bd62f-home\") pod \"prestop-hook-test-kserve-5f6b9b8bc6-9ksjx\" (UID: \"6a5121e0-0201-4024-bc90-6881d07bd62f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"
Sep 17 18:44:45.551989 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.551928    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a5121e0-0201-4024-bc90-6881d07bd62f-cabundle-cert\") pod \"prestop-hook-test-kserve-5f6b9b8bc6-9ksjx\" (UID: \"6a5121e0-0201-4024-bc90-6881d07bd62f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"
Sep 17 18:44:45.552162 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.552006    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a5121e0-0201-4024-bc90-6881d07bd62f-kserve-provision-location\") pod \"prestop-hook-test-kserve-5f6b9b8bc6-9ksjx\" (UID: \"6a5121e0-0201-4024-bc90-6881d07bd62f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"
Sep 17 18:44:45.552162 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.552034    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smg8p\" (UniqueName: \"kubernetes.io/projected/6a5121e0-0201-4024-bc90-6881d07bd62f-kube-api-access-smg8p\") pod \"prestop-hook-test-kserve-5f6b9b8bc6-9ksjx\" (UID: \"6a5121e0-0201-4024-bc90-6881d07bd62f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"
Sep 17 18:44:45.652679 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.652629    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smg8p\" (UniqueName: \"kubernetes.io/projected/6a5121e0-0201-4024-bc90-6881d07bd62f-kube-api-access-smg8p\") pod \"prestop-hook-test-kserve-5f6b9b8bc6-9ksjx\" (UID: \"6a5121e0-0201-4024-bc90-6881d07bd62f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"
Sep 17 18:44:45.652863 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.652700    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a5121e0-0201-4024-bc90-6881d07bd62f-dshm\") pod \"prestop-hook-test-kserve-5f6b9b8bc6-9ksjx\" (UID: \"6a5121e0-0201-4024-bc90-6881d07bd62f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"
Sep 17 18:44:45.652863 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.652723    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a5121e0-0201-4024-bc90-6881d07bd62f-tls-certs\") pod \"prestop-hook-test-kserve-5f6b9b8bc6-9ksjx\" (UID: \"6a5121e0-0201-4024-bc90-6881d07bd62f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"
Sep 17 18:44:45.652863 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.652740    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a5121e0-0201-4024-bc90-6881d07bd62f-tmp-dir\") pod \"prestop-hook-test-kserve-5f6b9b8bc6-9ksjx\" (UID: \"6a5121e0-0201-4024-bc90-6881d07bd62f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"
Sep 17 18:44:45.652863 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.652758    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a5121e0-0201-4024-bc90-6881d07bd62f-model-cache\") pod \"prestop-hook-test-kserve-5f6b9b8bc6-9ksjx\" (UID: \"6a5121e0-0201-4024-bc90-6881d07bd62f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"
Sep 17 18:44:45.652863 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.652779    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a5121e0-0201-4024-bc90-6881d07bd62f-home\") pod \"prestop-hook-test-kserve-5f6b9b8bc6-9ksjx\" (UID: \"6a5121e0-0201-4024-bc90-6881d07bd62f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"
Sep 17 18:44:45.652863 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.652814    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a5121e0-0201-4024-bc90-6881d07bd62f-cabundle-cert\") pod \"prestop-hook-test-kserve-5f6b9b8bc6-9ksjx\" (UID: \"6a5121e0-0201-4024-bc90-6881d07bd62f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"
Sep 17 18:44:45.653179 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.652869    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a5121e0-0201-4024-bc90-6881d07bd62f-kserve-provision-location\") pod \"prestop-hook-test-kserve-5f6b9b8bc6-9ksjx\" (UID: \"6a5121e0-0201-4024-bc90-6881d07bd62f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"
Sep 17 18:44:45.653179 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.653138    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a5121e0-0201-4024-bc90-6881d07bd62f-tmp-dir\") pod \"prestop-hook-test-kserve-5f6b9b8bc6-9ksjx\" (UID: \"6a5121e0-0201-4024-bc90-6881d07bd62f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"
Sep 17 18:44:45.653288 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.653190    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a5121e0-0201-4024-bc90-6881d07bd62f-model-cache\") pod \"prestop-hook-test-kserve-5f6b9b8bc6-9ksjx\" (UID: \"6a5121e0-0201-4024-bc90-6881d07bd62f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"
Sep 17 18:44:45.653288 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.653239    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a5121e0-0201-4024-bc90-6881d07bd62f-home\") pod \"prestop-hook-test-kserve-5f6b9b8bc6-9ksjx\" (UID: \"6a5121e0-0201-4024-bc90-6881d07bd62f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"
Sep 17 18:44:45.653370 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.653331    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a5121e0-0201-4024-bc90-6881d07bd62f-kserve-provision-location\") pod \"prestop-hook-test-kserve-5f6b9b8bc6-9ksjx\" (UID: \"6a5121e0-0201-4024-bc90-6881d07bd62f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"
Sep 17 18:44:45.653632 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.653609    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a5121e0-0201-4024-bc90-6881d07bd62f-cabundle-cert\") pod \"prestop-hook-test-kserve-5f6b9b8bc6-9ksjx\" (UID: \"6a5121e0-0201-4024-bc90-6881d07bd62f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"
Sep 17 18:44:45.655087 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.655066    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a5121e0-0201-4024-bc90-6881d07bd62f-dshm\") pod \"prestop-hook-test-kserve-5f6b9b8bc6-9ksjx\" (UID: \"6a5121e0-0201-4024-bc90-6881d07bd62f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"
Sep 17 18:44:45.655267 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.655251    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a5121e0-0201-4024-bc90-6881d07bd62f-tls-certs\") pod \"prestop-hook-test-kserve-5f6b9b8bc6-9ksjx\" (UID: \"6a5121e0-0201-4024-bc90-6881d07bd62f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"
Sep 17 18:44:45.676778 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.676749    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smg8p\" (UniqueName: \"kubernetes.io/projected/6a5121e0-0201-4024-bc90-6881d07bd62f-kube-api-access-smg8p\") pod \"prestop-hook-test-kserve-5f6b9b8bc6-9ksjx\" (UID: \"6a5121e0-0201-4024-bc90-6881d07bd62f\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"
Sep 17 18:44:45.742681 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:45.742585    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"
Sep 17 18:44:46.021044 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:46.020960    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"]
Sep 17 18:44:46.037384 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:44:46.037356    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5121e0_0201_4024_bc90_6881d07bd62f.slice/crio-8c60ab3851e1011bef7cbef81bf62d5fcd09e522ee5261e2a1d3ce0dee5ae922 WatchSource:0}: Error finding container 8c60ab3851e1011bef7cbef81bf62d5fcd09e522ee5261e2a1d3ce0dee5ae922: Status 404 returned error can't find the container with id 8c60ab3851e1011bef7cbef81bf62d5fcd09e522ee5261e2a1d3ce0dee5ae922
Sep 17 18:44:46.752175 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:46.752139    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx" event={"ID":"6a5121e0-0201-4024-bc90-6881d07bd62f","Type":"ContainerStarted","Data":"8566f637f2313800aac5e5eb04fa72a57d655498c42b2a0d4419ff9c1ee413b3"}
Sep 17 18:44:46.752607 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:46.752194    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx" event={"ID":"6a5121e0-0201-4024-bc90-6881d07bd62f","Type":"ContainerStarted","Data":"8c60ab3851e1011bef7cbef81bf62d5fcd09e522ee5261e2a1d3ce0dee5ae922"}
Sep 17 18:44:48.761995 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:48.761962    2583 generic.go:356] "Generic (PLEG): container finished" podID="6a5121e0-0201-4024-bc90-6881d07bd62f" containerID="8566f637f2313800aac5e5eb04fa72a57d655498c42b2a0d4419ff9c1ee413b3" exitCode=0
Sep 17 18:44:48.762399 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:48.762036    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx" event={"ID":"6a5121e0-0201-4024-bc90-6881d07bd62f","Type":"ContainerDied","Data":"8566f637f2313800aac5e5eb04fa72a57d655498c42b2a0d4419ff9c1ee413b3"}
Sep 17 18:44:49.769501 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:49.769464    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx" event={"ID":"6a5121e0-0201-4024-bc90-6881d07bd62f","Type":"ContainerStarted","Data":"0fcb81051aeeb7545926e6084d8128ea696ae3277015dbc387c3732f09478cb1"}
Sep 17 18:44:49.791596 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:49.791531    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx" podStartSLOduration=2.667546071 podStartE2EDuration="4.791512677s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.123966606s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:44:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:44:49.790211416 +0000 UTC m=+4200.881518212" watchObservedRunningTime="2026-09-17 18:44:49.791512677 +0000 UTC m=+4200.882819443"
Sep 17 18:44:50.743390 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:50.743351    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"
Sep 17 18:44:55.743327 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:55.743288    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"
Sep 17 18:44:55.744614 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:55.744587    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx" podUID="6a5121e0-0201-4024-bc90-6881d07bd62f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 17 18:44:57.097152 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:57.097114    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s"]
Sep 17 18:44:57.103653 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:57.103631    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s"
Sep 17 18:44:57.106868 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:57.106843    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-kgshk\""
Sep 17 18:44:57.107009 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:57.106842    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 17 18:44:57.107009 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:57.106986    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 17 18:44:57.107399 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:57.107375    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-9qw9c\""
Sep 17 18:44:57.107533 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:57.107513    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 17 18:44:57.112782 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:57.112763    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s"]
Sep 17 18:44:57.255528 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:57.255495    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg67f\" (UniqueName: \"kubernetes.io/projected/8136117c-6873-4a1a-a00a-a3dd8f2cd995-kube-api-access-lg67f\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s\" (UID: \"8136117c-6873-4a1a-a00a-a3dd8f2cd995\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s"
Sep 17 18:44:57.255722 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:57.255617    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8136117c-6873-4a1a-a00a-a3dd8f2cd995-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s\" (UID: \"8136117c-6873-4a1a-a00a-a3dd8f2cd995\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s"
Sep 17 18:44:57.356309 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:57.356223    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8136117c-6873-4a1a-a00a-a3dd8f2cd995-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s\" (UID: \"8136117c-6873-4a1a-a00a-a3dd8f2cd995\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s"
Sep 17 18:44:57.356309 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:57.356274    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg67f\" (UniqueName: \"kubernetes.io/projected/8136117c-6873-4a1a-a00a-a3dd8f2cd995-kube-api-access-lg67f\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s\" (UID: \"8136117c-6873-4a1a-a00a-a3dd8f2cd995\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s"
Sep 17 18:44:57.358753 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:57.358725    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8136117c-6873-4a1a-a00a-a3dd8f2cd995-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s\" (UID: \"8136117c-6873-4a1a-a00a-a3dd8f2cd995\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s"
Sep 17 18:44:57.364628 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:57.364603    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg67f\" (UniqueName: \"kubernetes.io/projected/8136117c-6873-4a1a-a00a-a3dd8f2cd995-kube-api-access-lg67f\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s\" (UID: \"8136117c-6873-4a1a-a00a-a3dd8f2cd995\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s"
Sep 17 18:44:57.416806 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:57.416772    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s"
Sep 17 18:44:57.691132 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:57.691060    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s"]
Sep 17 18:44:57.694359 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:44:57.694335    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8136117c_6873_4a1a_a00a_a3dd8f2cd995.slice/crio-c16d63b8c308957d48419de400f4e6989f70bb98cefa3edb7b475fbb22c63e86 WatchSource:0}: Error finding container c16d63b8c308957d48419de400f4e6989f70bb98cefa3edb7b475fbb22c63e86: Status 404 returned error can't find the container with id c16d63b8c308957d48419de400f4e6989f70bb98cefa3edb7b475fbb22c63e86
Sep 17 18:44:57.806469 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:57.806432    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s" event={"ID":"8136117c-6873-4a1a-a00a-a3dd8f2cd995","Type":"ContainerStarted","Data":"6861825ed0c0ad31e7552e85c502ba57f677daa2aab114989c698e7919ff1e26"}
Sep 17 18:44:57.806587 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:57.806502    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s" event={"ID":"8136117c-6873-4a1a-a00a-a3dd8f2cd995","Type":"ContainerStarted","Data":"c16d63b8c308957d48419de400f4e6989f70bb98cefa3edb7b475fbb22c63e86"}
Sep 17 18:44:57.806587 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:57.806566    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s"
Sep 17 18:44:57.826473 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:44:57.826400    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s" podStartSLOduration=0.826386604 podStartE2EDuration="826.386604ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:44:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:44:57.824679967 +0000 UTC m=+4208.915986747" watchObservedRunningTime="2026-09-17 18:44:57.826386604 +0000 UTC m=+4208.917693369"
Sep 17 18:45:05.743540 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:45:05.743445    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx" podUID="6a5121e0-0201-4024-bc90-6881d07bd62f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 17 18:45:15.743550 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:45:15.743510    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx" podUID="6a5121e0-0201-4024-bc90-6881d07bd62f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 17 18:45:25.743855 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:45:25.743807    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx" podUID="6a5121e0-0201-4024-bc90-6881d07bd62f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 17 18:45:28.812794 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:45:28.812756    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s"
Sep 17 18:45:30.233505 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:45:30.233469    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s"]
Sep 17 18:45:35.743937 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:45:35.743897    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx" podUID="6a5121e0-0201-4024-bc90-6881d07bd62f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 17 18:45:38.812575 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:45:38.812523    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s" podUID="8136117c-6873-4a1a-a00a-a3dd8f2cd995" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:45:45.253262 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:45:45.253211    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s" podUID="8136117c-6873-4a1a-a00a-a3dd8f2cd995" containerName="main" containerID="cri-o://6861825ed0c0ad31e7552e85c502ba57f677daa2aab114989c698e7919ff1e26" gracePeriod=45
Sep 17 18:45:45.743710 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:45:45.743675    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx" podUID="6a5121e0-0201-4024-bc90-6881d07bd62f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 17 18:45:48.811923 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:45:48.811885    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s" podUID="8136117c-6873-4a1a-a00a-a3dd8f2cd995" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:45:51.494826 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:45:51.494794    2583 scope.go:117] "RemoveContainer" containerID="c0729f852a07cb8cb9010bb8140875e17065302b6aea62cbee20bc628f073c4b"
Sep 17 18:45:55.743001 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:45:55.742957    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx" podUID="6a5121e0-0201-4024-bc90-6881d07bd62f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 17 18:45:58.812876 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:45:58.812837    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s" podUID="8136117c-6873-4a1a-a00a-a3dd8f2cd995" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:45:58.813369 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:45:58.812956    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s"
Sep 17 18:46:05.743783 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:05.743737    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx" podUID="6a5121e0-0201-4024-bc90-6881d07bd62f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 17 18:46:08.812434 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:08.812393    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s" podUID="8136117c-6873-4a1a-a00a-a3dd8f2cd995" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:46:15.500276 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:15.500244    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s"
Sep 17 18:46:15.612090 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:15.612062    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lg67f\" (UniqueName: \"kubernetes.io/projected/8136117c-6873-4a1a-a00a-a3dd8f2cd995-kube-api-access-lg67f\") pod \"8136117c-6873-4a1a-a00a-a3dd8f2cd995\" (UID: \"8136117c-6873-4a1a-a00a-a3dd8f2cd995\") "
Sep 17 18:46:15.612296 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:15.612116    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8136117c-6873-4a1a-a00a-a3dd8f2cd995-tls-certs\") pod \"8136117c-6873-4a1a-a00a-a3dd8f2cd995\" (UID: \"8136117c-6873-4a1a-a00a-a3dd8f2cd995\") "
Sep 17 18:46:15.614361 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:15.614326    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8136117c-6873-4a1a-a00a-a3dd8f2cd995-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8136117c-6873-4a1a-a00a-a3dd8f2cd995" (UID: "8136117c-6873-4a1a-a00a-a3dd8f2cd995"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:46:15.614466 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:15.614365    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8136117c-6873-4a1a-a00a-a3dd8f2cd995-kube-api-access-lg67f" (OuterVolumeSpecName: "kube-api-access-lg67f") pod "8136117c-6873-4a1a-a00a-a3dd8f2cd995" (UID: "8136117c-6873-4a1a-a00a-a3dd8f2cd995"). InnerVolumeSpecName "kube-api-access-lg67f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:46:15.713467 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:15.713425    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg67f\" (UniqueName: \"kubernetes.io/projected/8136117c-6873-4a1a-a00a-a3dd8f2cd995-kube-api-access-lg67f\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:46:15.713467 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:15.713465    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8136117c-6873-4a1a-a00a-a3dd8f2cd995-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:46:15.743772 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:15.743725    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx" podUID="6a5121e0-0201-4024-bc90-6881d07bd62f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 17 18:46:16.174315 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:16.174283    2583 generic.go:356] "Generic (PLEG): container finished" podID="8136117c-6873-4a1a-a00a-a3dd8f2cd995" containerID="6861825ed0c0ad31e7552e85c502ba57f677daa2aab114989c698e7919ff1e26" exitCode=0
Sep 17 18:46:16.174492 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:16.174341    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s" event={"ID":"8136117c-6873-4a1a-a00a-a3dd8f2cd995","Type":"ContainerDied","Data":"6861825ed0c0ad31e7552e85c502ba57f677daa2aab114989c698e7919ff1e26"}
Sep 17 18:46:16.174492 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:16.174351    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s"
Sep 17 18:46:16.174492 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:16.174386    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s" event={"ID":"8136117c-6873-4a1a-a00a-a3dd8f2cd995","Type":"ContainerDied","Data":"c16d63b8c308957d48419de400f4e6989f70bb98cefa3edb7b475fbb22c63e86"}
Sep 17 18:46:16.174492 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:16.174409    2583 scope.go:117] "RemoveContainer" containerID="6861825ed0c0ad31e7552e85c502ba57f677daa2aab114989c698e7919ff1e26"
Sep 17 18:46:16.182679 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:16.182643    2583 scope.go:117] "RemoveContainer" containerID="6861825ed0c0ad31e7552e85c502ba57f677daa2aab114989c698e7919ff1e26"
Sep 17 18:46:16.182941 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:46:16.182921    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6861825ed0c0ad31e7552e85c502ba57f677daa2aab114989c698e7919ff1e26\": container with ID starting with 6861825ed0c0ad31e7552e85c502ba57f677daa2aab114989c698e7919ff1e26 not found: ID does not exist" containerID="6861825ed0c0ad31e7552e85c502ba57f677daa2aab114989c698e7919ff1e26"
Sep 17 18:46:16.183013 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:16.182949    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6861825ed0c0ad31e7552e85c502ba57f677daa2aab114989c698e7919ff1e26"} err="failed to get container status \"6861825ed0c0ad31e7552e85c502ba57f677daa2aab114989c698e7919ff1e26\": rpc error: code = NotFound desc = could not find container \"6861825ed0c0ad31e7552e85c502ba57f677daa2aab114989c698e7919ff1e26\": container with ID starting with 6861825ed0c0ad31e7552e85c502ba57f677daa2aab114989c698e7919ff1e26 not found: ID does not exist"
Sep 17 18:46:16.207460 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:16.207433    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s"]
Sep 17 18:46:16.210825 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:16.210804    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schex4w8s"]
Sep 17 18:46:17.040402 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:17.040368    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8136117c-6873-4a1a-a00a-a3dd8f2cd995" path="/var/lib/kubelet/pods/8136117c-6873-4a1a-a00a-a3dd8f2cd995/volumes"
Sep 17 18:46:25.743135 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:25.743090    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx" podUID="6a5121e0-0201-4024-bc90-6881d07bd62f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 17 18:46:35.743107 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:35.743061    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx" podUID="6a5121e0-0201-4024-bc90-6881d07bd62f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Sep 17 18:46:42.048797 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:42.048760    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h"]
Sep 17 18:46:42.049405 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:42.049073    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8136117c-6873-4a1a-a00a-a3dd8f2cd995" containerName="main"
Sep 17 18:46:42.049405 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:42.049084    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8136117c-6873-4a1a-a00a-a3dd8f2cd995" containerName="main"
Sep 17 18:46:42.049405 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:42.049151    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8136117c-6873-4a1a-a00a-a3dd8f2cd995" containerName="main"
Sep 17 18:46:42.053181 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:42.053161    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h"
Sep 17 18:46:42.056532 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:42.056509    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 17 18:46:42.056689 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:42.056517    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-ffzf7\""
Sep 17 18:46:42.056689 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:42.056519    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 17 18:46:42.056689 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:42.056603    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 17 18:46:42.056689 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:42.056545    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-t9rz6\""
Sep 17 18:46:42.063992 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:42.063969    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h"]
Sep 17 18:46:42.233191 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:42.233153    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwgd6\" (UniqueName: \"kubernetes.io/projected/85ba420f-2598-47ae-8825-d62a20548390-kube-api-access-jwgd6\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h\" (UID: \"85ba420f-2598-47ae-8825-d62a20548390\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h"
Sep 17 18:46:42.233191 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:42.233191    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85ba420f-2598-47ae-8825-d62a20548390-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h\" (UID: \"85ba420f-2598-47ae-8825-d62a20548390\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h"
Sep 17 18:46:42.333874 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:42.333792    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwgd6\" (UniqueName: \"kubernetes.io/projected/85ba420f-2598-47ae-8825-d62a20548390-kube-api-access-jwgd6\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h\" (UID: \"85ba420f-2598-47ae-8825-d62a20548390\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h"
Sep 17 18:46:42.333874 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:42.333830    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85ba420f-2598-47ae-8825-d62a20548390-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h\" (UID: \"85ba420f-2598-47ae-8825-d62a20548390\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h"
Sep 17 18:46:42.336394 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:42.336365    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85ba420f-2598-47ae-8825-d62a20548390-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h\" (UID: \"85ba420f-2598-47ae-8825-d62a20548390\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h"
Sep 17 18:46:42.341969 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:42.341949    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwgd6\" (UniqueName: \"kubernetes.io/projected/85ba420f-2598-47ae-8825-d62a20548390-kube-api-access-jwgd6\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h\" (UID: \"85ba420f-2598-47ae-8825-d62a20548390\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h"
Sep 17 18:46:42.365922 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:42.365884    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h"
Sep 17 18:46:42.649451 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:42.649413    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h"]
Sep 17 18:46:42.655511 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:46:42.655484    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ba420f_2598_47ae_8825_d62a20548390.slice/crio-a19061c7e7857a323ff8b9351735c709e31159b51d3ea9a31c9c6ba08db7c53c WatchSource:0}: Error finding container a19061c7e7857a323ff8b9351735c709e31159b51d3ea9a31c9c6ba08db7c53c: Status 404 returned error can't find the container with id a19061c7e7857a323ff8b9351735c709e31159b51d3ea9a31c9c6ba08db7c53c
Sep 17 18:46:43.292904 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:43.292860    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h" event={"ID":"85ba420f-2598-47ae-8825-d62a20548390","Type":"ContainerStarted","Data":"5d5874848ff88571f18cf4675094e4bb53596341231ee2382343e1d389583e48"}
Sep 17 18:46:43.293318 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:43.292921    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h" event={"ID":"85ba420f-2598-47ae-8825-d62a20548390","Type":"ContainerStarted","Data":"a19061c7e7857a323ff8b9351735c709e31159b51d3ea9a31c9c6ba08db7c53c"}
Sep 17 18:46:43.293318 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:43.292949    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h"
Sep 17 18:46:43.310940 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:43.310894    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h" podStartSLOduration=1.310879364 podStartE2EDuration="1.310879364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:46:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:46:43.309384381 +0000 UTC m=+4314.400691164" watchObservedRunningTime="2026-09-17 18:46:43.310879364 +0000 UTC m=+4314.402186130"
Sep 17 18:46:45.753335 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:45.753305    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"
Sep 17 18:46:45.761479 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:45.761454    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"
Sep 17 18:46:49.904844 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:46:49.904808    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"]
Sep 17 18:47:04.926945 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:04.926888    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx" podUID="6a5121e0-0201-4024-bc90-6881d07bd62f" containerName="main" containerID="cri-o://0fcb81051aeeb7545926e6084d8128ea696ae3277015dbc387c3732f09478cb1" gracePeriod=45
Sep 17 18:47:05.592905 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:05.592879    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"
Sep 17 18:47:05.622036 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:05.622007    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a5121e0-0201-4024-bc90-6881d07bd62f-kserve-provision-location\") pod \"6a5121e0-0201-4024-bc90-6881d07bd62f\" (UID: \"6a5121e0-0201-4024-bc90-6881d07bd62f\") "
Sep 17 18:47:05.622191 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:05.622054    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smg8p\" (UniqueName: \"kubernetes.io/projected/6a5121e0-0201-4024-bc90-6881d07bd62f-kube-api-access-smg8p\") pod \"6a5121e0-0201-4024-bc90-6881d07bd62f\" (UID: \"6a5121e0-0201-4024-bc90-6881d07bd62f\") "
Sep 17 18:47:05.622191 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:05.622084    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a5121e0-0201-4024-bc90-6881d07bd62f-dshm\") pod \"6a5121e0-0201-4024-bc90-6881d07bd62f\" (UID: \"6a5121e0-0201-4024-bc90-6881d07bd62f\") "
Sep 17 18:47:05.622191 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:05.622104    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a5121e0-0201-4024-bc90-6881d07bd62f-model-cache\") pod \"6a5121e0-0201-4024-bc90-6881d07bd62f\" (UID: \"6a5121e0-0201-4024-bc90-6881d07bd62f\") "
Sep 17 18:47:05.622191 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:05.622117    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a5121e0-0201-4024-bc90-6881d07bd62f-home\") pod \"6a5121e0-0201-4024-bc90-6881d07bd62f\" (UID: \"6a5121e0-0201-4024-bc90-6881d07bd62f\") "
Sep 17 18:47:05.622191 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:05.622141    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a5121e0-0201-4024-bc90-6881d07bd62f-cabundle-cert\") pod \"6a5121e0-0201-4024-bc90-6881d07bd62f\" (UID: \"6a5121e0-0201-4024-bc90-6881d07bd62f\") "
Sep 17 18:47:05.622191 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:05.622164    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a5121e0-0201-4024-bc90-6881d07bd62f-tmp-dir\") pod \"6a5121e0-0201-4024-bc90-6881d07bd62f\" (UID: \"6a5121e0-0201-4024-bc90-6881d07bd62f\") "
Sep 17 18:47:05.622502 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:05.622219    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a5121e0-0201-4024-bc90-6881d07bd62f-tls-certs\") pod \"6a5121e0-0201-4024-bc90-6881d07bd62f\" (UID: \"6a5121e0-0201-4024-bc90-6881d07bd62f\") "
Sep 17 18:47:05.622563 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:05.622498    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a5121e0-0201-4024-bc90-6881d07bd62f-model-cache" (OuterVolumeSpecName: "model-cache") pod "6a5121e0-0201-4024-bc90-6881d07bd62f" (UID: "6a5121e0-0201-4024-bc90-6881d07bd62f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:47:05.622853 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:05.622810    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a5121e0-0201-4024-bc90-6881d07bd62f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6a5121e0-0201-4024-bc90-6881d07bd62f" (UID: "6a5121e0-0201-4024-bc90-6881d07bd62f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:47:05.622962 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:05.622878    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a5121e0-0201-4024-bc90-6881d07bd62f-home" (OuterVolumeSpecName: "home") pod "6a5121e0-0201-4024-bc90-6881d07bd62f" (UID: "6a5121e0-0201-4024-bc90-6881d07bd62f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:47:05.625137 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:05.625094    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a5121e0-0201-4024-bc90-6881d07bd62f-dshm" (OuterVolumeSpecName: "dshm") pod "6a5121e0-0201-4024-bc90-6881d07bd62f" (UID: "6a5121e0-0201-4024-bc90-6881d07bd62f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:47:05.626335 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:05.626297    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a5121e0-0201-4024-bc90-6881d07bd62f-kube-api-access-smg8p" (OuterVolumeSpecName: "kube-api-access-smg8p") pod "6a5121e0-0201-4024-bc90-6881d07bd62f" (UID: "6a5121e0-0201-4024-bc90-6881d07bd62f"). InnerVolumeSpecName "kube-api-access-smg8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:47:05.632779 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:05.632751    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a5121e0-0201-4024-bc90-6881d07bd62f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a5121e0-0201-4024-bc90-6881d07bd62f" (UID: "6a5121e0-0201-4024-bc90-6881d07bd62f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:47:05.641261 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:05.641225    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a5121e0-0201-4024-bc90-6881d07bd62f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6a5121e0-0201-4024-bc90-6881d07bd62f" (UID: "6a5121e0-0201-4024-bc90-6881d07bd62f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:47:05.650105 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:05.650076    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a5121e0-0201-4024-bc90-6881d07bd62f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6a5121e0-0201-4024-bc90-6881d07bd62f" (UID: "6a5121e0-0201-4024-bc90-6881d07bd62f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:47:05.723512 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:05.723425    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a5121e0-0201-4024-bc90-6881d07bd62f-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:47:05.723512 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:05.723463    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6a5121e0-0201-4024-bc90-6881d07bd62f-kserve-provision-location\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:47:05.723512 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:05.723474    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smg8p\" (UniqueName: \"kubernetes.io/projected/6a5121e0-0201-4024-bc90-6881d07bd62f-kube-api-access-smg8p\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:47:05.723512 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:05.723482    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a5121e0-0201-4024-bc90-6881d07bd62f-dshm\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:47:05.723512 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:05.723491    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a5121e0-0201-4024-bc90-6881d07bd62f-model-cache\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:47:05.723512 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:05.723499    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a5121e0-0201-4024-bc90-6881d07bd62f-home\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:47:05.723512 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:05.723506    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6a5121e0-0201-4024-bc90-6881d07bd62f-cabundle-cert\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:47:05.723512 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:05.723513    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a5121e0-0201-4024-bc90-6881d07bd62f-tmp-dir\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:47:06.396260 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:06.396222    2583 generic.go:356] "Generic (PLEG): container finished" podID="6a5121e0-0201-4024-bc90-6881d07bd62f" containerID="0fcb81051aeeb7545926e6084d8128ea696ae3277015dbc387c3732f09478cb1" exitCode=0
Sep 17 18:47:06.396701 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:06.396274    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"
Sep 17 18:47:06.396701 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:06.396310    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx" event={"ID":"6a5121e0-0201-4024-bc90-6881d07bd62f","Type":"ContainerDied","Data":"0fcb81051aeeb7545926e6084d8128ea696ae3277015dbc387c3732f09478cb1"}
Sep 17 18:47:06.396701 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:06.396383    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx" event={"ID":"6a5121e0-0201-4024-bc90-6881d07bd62f","Type":"ContainerDied","Data":"8c60ab3851e1011bef7cbef81bf62d5fcd09e522ee5261e2a1d3ce0dee5ae922"}
Sep 17 18:47:06.396701 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:06.396406    2583 scope.go:117] "RemoveContainer" containerID="0fcb81051aeeb7545926e6084d8128ea696ae3277015dbc387c3732f09478cb1"
Sep 17 18:47:06.405302 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:06.405286    2583 scope.go:117] "RemoveContainer" containerID="8566f637f2313800aac5e5eb04fa72a57d655498c42b2a0d4419ff9c1ee413b3"
Sep 17 18:47:06.412565 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:06.412548    2583 scope.go:117] "RemoveContainer" containerID="0fcb81051aeeb7545926e6084d8128ea696ae3277015dbc387c3732f09478cb1"
Sep 17 18:47:06.412838 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:47:06.412819    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fcb81051aeeb7545926e6084d8128ea696ae3277015dbc387c3732f09478cb1\": container with ID starting with 0fcb81051aeeb7545926e6084d8128ea696ae3277015dbc387c3732f09478cb1 not found: ID does not exist" containerID="0fcb81051aeeb7545926e6084d8128ea696ae3277015dbc387c3732f09478cb1"
Sep 17 18:47:06.412918 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:06.412850    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fcb81051aeeb7545926e6084d8128ea696ae3277015dbc387c3732f09478cb1"} err="failed to get container status \"0fcb81051aeeb7545926e6084d8128ea696ae3277015dbc387c3732f09478cb1\": rpc error: code = NotFound desc = could not find container \"0fcb81051aeeb7545926e6084d8128ea696ae3277015dbc387c3732f09478cb1\": container with ID starting with 0fcb81051aeeb7545926e6084d8128ea696ae3277015dbc387c3732f09478cb1 not found: ID does not exist"
Sep 17 18:47:06.412918 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:06.412873    2583 scope.go:117] "RemoveContainer" containerID="8566f637f2313800aac5e5eb04fa72a57d655498c42b2a0d4419ff9c1ee413b3"
Sep 17 18:47:06.413111 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:47:06.413095    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8566f637f2313800aac5e5eb04fa72a57d655498c42b2a0d4419ff9c1ee413b3\": container with ID starting with 8566f637f2313800aac5e5eb04fa72a57d655498c42b2a0d4419ff9c1ee413b3 not found: ID does not exist" containerID="8566f637f2313800aac5e5eb04fa72a57d655498c42b2a0d4419ff9c1ee413b3"
Sep 17 18:47:06.413149 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:06.413116    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8566f637f2313800aac5e5eb04fa72a57d655498c42b2a0d4419ff9c1ee413b3"} err="failed to get container status \"8566f637f2313800aac5e5eb04fa72a57d655498c42b2a0d4419ff9c1ee413b3\": rpc error: code = NotFound desc = could not find container \"8566f637f2313800aac5e5eb04fa72a57d655498c42b2a0d4419ff9c1ee413b3\": container with ID starting with 8566f637f2313800aac5e5eb04fa72a57d655498c42b2a0d4419ff9c1ee413b3 not found: ID does not exist"
Sep 17 18:47:06.419581 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:06.419559    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"]
Sep 17 18:47:06.426277 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:06.426252    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f6b9b8bc6-9ksjx"]
Sep 17 18:47:07.040344 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:07.040311    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a5121e0-0201-4024-bc90-6881d07bd62f" path="/var/lib/kubelet/pods/6a5121e0-0201-4024-bc90-6881d07bd62f/volumes"
Sep 17 18:47:14.299223 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:14.299195    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h"
Sep 17 18:47:17.169986 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:17.169952    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h"]
Sep 17 18:47:24.298654 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:24.298610    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h" podUID="85ba420f-2598-47ae-8825-d62a20548390" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:47:32.189161 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:32.189120    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h" podUID="85ba420f-2598-47ae-8825-d62a20548390" containerName="main" containerID="cri-o://5d5874848ff88571f18cf4675094e4bb53596341231ee2382343e1d389583e48" gracePeriod=45
Sep 17 18:47:34.299185 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:34.299141    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h" podUID="85ba420f-2598-47ae-8825-d62a20548390" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:47:44.298726 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:44.298688    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h" podUID="85ba420f-2598-47ae-8825-d62a20548390" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:47:44.299148 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:44.298789    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h"
Sep 17 18:47:54.298779 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:47:54.298740    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h" podUID="85ba420f-2598-47ae-8825-d62a20548390" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:48:02.424040 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:02.424007    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h"
Sep 17 18:48:02.451254 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:02.451178    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85ba420f-2598-47ae-8825-d62a20548390-tls-certs\") pod \"85ba420f-2598-47ae-8825-d62a20548390\" (UID: \"85ba420f-2598-47ae-8825-d62a20548390\") "
Sep 17 18:48:02.451418 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:02.451279    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwgd6\" (UniqueName: \"kubernetes.io/projected/85ba420f-2598-47ae-8825-d62a20548390-kube-api-access-jwgd6\") pod \"85ba420f-2598-47ae-8825-d62a20548390\" (UID: \"85ba420f-2598-47ae-8825-d62a20548390\") "
Sep 17 18:48:02.453696 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:02.453641    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85ba420f-2598-47ae-8825-d62a20548390-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "85ba420f-2598-47ae-8825-d62a20548390" (UID: "85ba420f-2598-47ae-8825-d62a20548390"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:48:02.453800 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:02.453706    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85ba420f-2598-47ae-8825-d62a20548390-kube-api-access-jwgd6" (OuterVolumeSpecName: "kube-api-access-jwgd6") pod "85ba420f-2598-47ae-8825-d62a20548390" (UID: "85ba420f-2598-47ae-8825-d62a20548390"). InnerVolumeSpecName "kube-api-access-jwgd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:48:02.552259 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:02.552218    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwgd6\" (UniqueName: \"kubernetes.io/projected/85ba420f-2598-47ae-8825-d62a20548390-kube-api-access-jwgd6\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:48:02.552259 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:02.552250    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85ba420f-2598-47ae-8825-d62a20548390-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:48:02.629489 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:02.629453    2583 generic.go:356] "Generic (PLEG): container finished" podID="85ba420f-2598-47ae-8825-d62a20548390" containerID="5d5874848ff88571f18cf4675094e4bb53596341231ee2382343e1d389583e48" exitCode=0
Sep 17 18:48:02.629685 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:02.629509    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h"
Sep 17 18:48:02.629685 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:02.629521    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h" event={"ID":"85ba420f-2598-47ae-8825-d62a20548390","Type":"ContainerDied","Data":"5d5874848ff88571f18cf4675094e4bb53596341231ee2382343e1d389583e48"}
Sep 17 18:48:02.629685 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:02.629594    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h" event={"ID":"85ba420f-2598-47ae-8825-d62a20548390","Type":"ContainerDied","Data":"a19061c7e7857a323ff8b9351735c709e31159b51d3ea9a31c9c6ba08db7c53c"}
Sep 17 18:48:02.629685 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:02.629622    2583 scope.go:117] "RemoveContainer" containerID="5d5874848ff88571f18cf4675094e4bb53596341231ee2382343e1d389583e48"
Sep 17 18:48:02.638203 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:02.638180    2583 scope.go:117] "RemoveContainer" containerID="5d5874848ff88571f18cf4675094e4bb53596341231ee2382343e1d389583e48"
Sep 17 18:48:02.638499 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:48:02.638480    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d5874848ff88571f18cf4675094e4bb53596341231ee2382343e1d389583e48\": container with ID starting with 5d5874848ff88571f18cf4675094e4bb53596341231ee2382343e1d389583e48 not found: ID does not exist" containerID="5d5874848ff88571f18cf4675094e4bb53596341231ee2382343e1d389583e48"
Sep 17 18:48:02.638558 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:02.638508    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d5874848ff88571f18cf4675094e4bb53596341231ee2382343e1d389583e48"} err="failed to get container status \"5d5874848ff88571f18cf4675094e4bb53596341231ee2382343e1d389583e48\": rpc error: code = NotFound desc = could not find container \"5d5874848ff88571f18cf4675094e4bb53596341231ee2382343e1d389583e48\": container with ID starting with 5d5874848ff88571f18cf4675094e4bb53596341231ee2382343e1d389583e48 not found: ID does not exist"
Sep 17 18:48:02.651404 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:02.651379    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h"]
Sep 17 18:48:02.657407 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:02.657384    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c97qm9h"]
Sep 17 18:48:03.040633 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:03.040598    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85ba420f-2598-47ae-8825-d62a20548390" path="/var/lib/kubelet/pods/85ba420f-2598-47ae-8825-d62a20548390/volumes"
Sep 17 18:48:42.635326 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:42.635290    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8"]
Sep 17 18:48:42.635944 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:42.635584    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a5121e0-0201-4024-bc90-6881d07bd62f" containerName="storage-initializer"
Sep 17 18:48:42.635944 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:42.635596    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a5121e0-0201-4024-bc90-6881d07bd62f" containerName="storage-initializer"
Sep 17 18:48:42.635944 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:42.635605    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85ba420f-2598-47ae-8825-d62a20548390" containerName="main"
Sep 17 18:48:42.635944 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:42.635610    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="85ba420f-2598-47ae-8825-d62a20548390" containerName="main"
Sep 17 18:48:42.635944 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:42.635619    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a5121e0-0201-4024-bc90-6881d07bd62f" containerName="main"
Sep 17 18:48:42.635944 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:42.635624    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a5121e0-0201-4024-bc90-6881d07bd62f" containerName="main"
Sep 17 18:48:42.635944 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:42.635686    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="85ba420f-2598-47ae-8825-d62a20548390" containerName="main"
Sep 17 18:48:42.635944 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:42.635696    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a5121e0-0201-4024-bc90-6881d07bd62f" containerName="main"
Sep 17 18:48:42.638431 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:42.638413    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8"
Sep 17 18:48:42.641742 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:42.641682    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 17 18:48:42.641916 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:42.641717    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-c8knr\""
Sep 17 18:48:42.641916 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:42.641906    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-8ttfc\""
Sep 17 18:48:42.642035 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:42.641987    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 17 18:48:42.642101 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:42.642033    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 17 18:48:42.649013 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:42.648993    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8"]
Sep 17 18:48:42.742005 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:42.741973    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qplm\" (UniqueName: \"kubernetes.io/projected/6a0928f2-09a4-4ce6-90a0-ca2b0656a767-kube-api-access-2qplm\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8\" (UID: \"6a0928f2-09a4-4ce6-90a0-ca2b0656a767\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8"
Sep 17 18:48:42.742173 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:42.742026    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a0928f2-09a4-4ce6-90a0-ca2b0656a767-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8\" (UID: \"6a0928f2-09a4-4ce6-90a0-ca2b0656a767\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8"
Sep 17 18:48:42.842958 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:42.842925    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qplm\" (UniqueName: \"kubernetes.io/projected/6a0928f2-09a4-4ce6-90a0-ca2b0656a767-kube-api-access-2qplm\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8\" (UID: \"6a0928f2-09a4-4ce6-90a0-ca2b0656a767\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8"
Sep 17 18:48:42.843116 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:42.842992    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a0928f2-09a4-4ce6-90a0-ca2b0656a767-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8\" (UID: \"6a0928f2-09a4-4ce6-90a0-ca2b0656a767\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8"
Sep 17 18:48:42.845624 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:42.845592    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a0928f2-09a4-4ce6-90a0-ca2b0656a767-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8\" (UID: \"6a0928f2-09a4-4ce6-90a0-ca2b0656a767\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8"
Sep 17 18:48:42.850884 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:42.850855    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qplm\" (UniqueName: \"kubernetes.io/projected/6a0928f2-09a4-4ce6-90a0-ca2b0656a767-kube-api-access-2qplm\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8\" (UID: \"6a0928f2-09a4-4ce6-90a0-ca2b0656a767\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8"
Sep 17 18:48:42.948770 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:42.948689    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8"
Sep 17 18:48:43.219653 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:43.219573    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8"]
Sep 17 18:48:43.223003 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:48:43.222976    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0928f2_09a4_4ce6_90a0_ca2b0656a767.slice/crio-9701fb5c816d49c121cd27fd94241e32cb557a2353b56f31abdeb13ebe880d67 WatchSource:0}: Error finding container 9701fb5c816d49c121cd27fd94241e32cb557a2353b56f31abdeb13ebe880d67: Status 404 returned error can't find the container with id 9701fb5c816d49c121cd27fd94241e32cb557a2353b56f31abdeb13ebe880d67
Sep 17 18:48:43.795778 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:43.795744    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8" event={"ID":"6a0928f2-09a4-4ce6-90a0-ca2b0656a767","Type":"ContainerStarted","Data":"d2df79b30b7c4ff67caf20e660c538cf06ac3ec5b5678f4fb447f24a96c41d56"}
Sep 17 18:48:43.796235 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:43.795791    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8" event={"ID":"6a0928f2-09a4-4ce6-90a0-ca2b0656a767","Type":"ContainerStarted","Data":"9701fb5c816d49c121cd27fd94241e32cb557a2353b56f31abdeb13ebe880d67"}
Sep 17 18:48:43.796235 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:43.795829    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8"
Sep 17 18:48:43.814753 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:48:43.814702    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8" podStartSLOduration=1.814688656 podStartE2EDuration="1.814688656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:48:43.812742914 +0000 UTC m=+4434.904049690" watchObservedRunningTime="2026-09-17 18:48:43.814688656 +0000 UTC m=+4434.905995450"
Sep 17 18:49:14.801670 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:49:14.801639    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8"
Sep 17 18:49:40.114532 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:49:40.114492    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8"]
Sep 17 18:49:44.801683 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:49:44.801624    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8" podUID="6a0928f2-09a4-4ce6-90a0-ca2b0656a767" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:49:54.801786 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:49:54.801743    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8" podUID="6a0928f2-09a4-4ce6-90a0-ca2b0656a767" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:49:55.133965 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:49:55.133931    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8" podUID="6a0928f2-09a4-4ce6-90a0-ca2b0656a767" containerName="main" containerID="cri-o://d2df79b30b7c4ff67caf20e660c538cf06ac3ec5b5678f4fb447f24a96c41d56" gracePeriod=45
Sep 17 18:50:04.801559 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:04.801519    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8" podUID="6a0928f2-09a4-4ce6-90a0-ca2b0656a767" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:50:04.802001 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:04.801627    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8"
Sep 17 18:50:14.801721 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:14.801681    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8" podUID="6a0928f2-09a4-4ce6-90a0-ca2b0656a767" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:50:24.802373 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:24.802329    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8" podUID="6a0928f2-09a4-4ce6-90a0-ca2b0656a767" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 17 18:50:25.375072 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:25.375048    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8"
Sep 17 18:50:25.502014 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:25.501945    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a0928f2-09a4-4ce6-90a0-ca2b0656a767-tls-certs\") pod \"6a0928f2-09a4-4ce6-90a0-ca2b0656a767\" (UID: \"6a0928f2-09a4-4ce6-90a0-ca2b0656a767\") "
Sep 17 18:50:25.502014 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:25.501995    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qplm\" (UniqueName: \"kubernetes.io/projected/6a0928f2-09a4-4ce6-90a0-ca2b0656a767-kube-api-access-2qplm\") pod \"6a0928f2-09a4-4ce6-90a0-ca2b0656a767\" (UID: \"6a0928f2-09a4-4ce6-90a0-ca2b0656a767\") "
Sep 17 18:50:25.504202 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:25.504171    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a0928f2-09a4-4ce6-90a0-ca2b0656a767-kube-api-access-2qplm" (OuterVolumeSpecName: "kube-api-access-2qplm") pod "6a0928f2-09a4-4ce6-90a0-ca2b0656a767" (UID: "6a0928f2-09a4-4ce6-90a0-ca2b0656a767"). InnerVolumeSpecName "kube-api-access-2qplm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:50:25.504265 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:25.504187    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a0928f2-09a4-4ce6-90a0-ca2b0656a767-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a0928f2-09a4-4ce6-90a0-ca2b0656a767" (UID: "6a0928f2-09a4-4ce6-90a0-ca2b0656a767"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:50:25.603140 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:25.603111    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a0928f2-09a4-4ce6-90a0-ca2b0656a767-tls-certs\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:50:25.603140 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:25.603137    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qplm\" (UniqueName: \"kubernetes.io/projected/6a0928f2-09a4-4ce6-90a0-ca2b0656a767-kube-api-access-2qplm\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:50:26.198219 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:26.198179    2583 generic.go:356] "Generic (PLEG): container finished" podID="6a0928f2-09a4-4ce6-90a0-ca2b0656a767" containerID="d2df79b30b7c4ff67caf20e660c538cf06ac3ec5b5678f4fb447f24a96c41d56" exitCode=0
Sep 17 18:50:26.198654 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:26.198247    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8"
Sep 17 18:50:26.198654 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:26.198258    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8" event={"ID":"6a0928f2-09a4-4ce6-90a0-ca2b0656a767","Type":"ContainerDied","Data":"d2df79b30b7c4ff67caf20e660c538cf06ac3ec5b5678f4fb447f24a96c41d56"}
Sep 17 18:50:26.198654 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:26.198313    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8" event={"ID":"6a0928f2-09a4-4ce6-90a0-ca2b0656a767","Type":"ContainerDied","Data":"9701fb5c816d49c121cd27fd94241e32cb557a2353b56f31abdeb13ebe880d67"}
Sep 17 18:50:26.198654 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:26.198335    2583 scope.go:117] "RemoveContainer" containerID="d2df79b30b7c4ff67caf20e660c538cf06ac3ec5b5678f4fb447f24a96c41d56"
Sep 17 18:50:26.206568 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:26.206554    2583 scope.go:117] "RemoveContainer" containerID="d2df79b30b7c4ff67caf20e660c538cf06ac3ec5b5678f4fb447f24a96c41d56"
Sep 17 18:50:26.206849 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:50:26.206828    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2df79b30b7c4ff67caf20e660c538cf06ac3ec5b5678f4fb447f24a96c41d56\": container with ID starting with d2df79b30b7c4ff67caf20e660c538cf06ac3ec5b5678f4fb447f24a96c41d56 not found: ID does not exist" containerID="d2df79b30b7c4ff67caf20e660c538cf06ac3ec5b5678f4fb447f24a96c41d56"
Sep 17 18:50:26.206930 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:26.206855    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2df79b30b7c4ff67caf20e660c538cf06ac3ec5b5678f4fb447f24a96c41d56"} err="failed to get container status \"d2df79b30b7c4ff67caf20e660c538cf06ac3ec5b5678f4fb447f24a96c41d56\": rpc error: code = NotFound desc = could not find container \"d2df79b30b7c4ff67caf20e660c538cf06ac3ec5b5678f4fb447f24a96c41d56\": container with ID starting with d2df79b30b7c4ff67caf20e660c538cf06ac3ec5b5678f4fb447f24a96c41d56 not found: ID does not exist"
Sep 17 18:50:26.219761 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:26.219732    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8"]
Sep 17 18:50:26.223438 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:26.223415    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-d7nw8"]
Sep 17 18:50:27.039859 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:27.039823    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a0928f2-09a4-4ce6-90a0-ca2b0656a767" path="/var/lib/kubelet/pods/6a0928f2-09a4-4ce6-90a0-ca2b0656a767/volumes"
Sep 17 18:50:30.689988 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:30.689959    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rkfh7/must-gather-nz7qt"]
Sep 17 18:50:30.690398 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:30.690221    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a0928f2-09a4-4ce6-90a0-ca2b0656a767" containerName="main"
Sep 17 18:50:30.690398 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:30.690231    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a0928f2-09a4-4ce6-90a0-ca2b0656a767" containerName="main"
Sep 17 18:50:30.690398 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:30.690290    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a0928f2-09a4-4ce6-90a0-ca2b0656a767" containerName="main"
Sep 17 18:50:30.694478 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:30.694459    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rkfh7/must-gather-nz7qt"
Sep 17 18:50:30.696879 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:30.696859    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rkfh7\"/\"kube-root-ca.crt\""
Sep 17 18:50:30.697016 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:30.696996    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rkfh7\"/\"default-dockercfg-6fs5g\""
Sep 17 18:50:30.697858 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:30.697842    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rkfh7\"/\"openshift-service-ca.crt\""
Sep 17 18:50:30.703947 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:30.703927    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rkfh7/must-gather-nz7qt"]
Sep 17 18:50:30.845353 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:30.845308    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8wdr\" (UniqueName: \"kubernetes.io/projected/2f9542fb-b8fb-42f0-9411-e77a6e51ec36-kube-api-access-t8wdr\") pod \"must-gather-nz7qt\" (UID: \"2f9542fb-b8fb-42f0-9411-e77a6e51ec36\") " pod="openshift-must-gather-rkfh7/must-gather-nz7qt"
Sep 17 18:50:30.845353 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:30.845354    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2f9542fb-b8fb-42f0-9411-e77a6e51ec36-must-gather-output\") pod \"must-gather-nz7qt\" (UID: \"2f9542fb-b8fb-42f0-9411-e77a6e51ec36\") " pod="openshift-must-gather-rkfh7/must-gather-nz7qt"
Sep 17 18:50:30.946217 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:30.946121    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8wdr\" (UniqueName: \"kubernetes.io/projected/2f9542fb-b8fb-42f0-9411-e77a6e51ec36-kube-api-access-t8wdr\") pod \"must-gather-nz7qt\" (UID: \"2f9542fb-b8fb-42f0-9411-e77a6e51ec36\") " pod="openshift-must-gather-rkfh7/must-gather-nz7qt"
Sep 17 18:50:30.946217 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:30.946165    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2f9542fb-b8fb-42f0-9411-e77a6e51ec36-must-gather-output\") pod \"must-gather-nz7qt\" (UID: \"2f9542fb-b8fb-42f0-9411-e77a6e51ec36\") " pod="openshift-must-gather-rkfh7/must-gather-nz7qt"
Sep 17 18:50:30.946457 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:30.946430    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2f9542fb-b8fb-42f0-9411-e77a6e51ec36-must-gather-output\") pod \"must-gather-nz7qt\" (UID: \"2f9542fb-b8fb-42f0-9411-e77a6e51ec36\") " pod="openshift-must-gather-rkfh7/must-gather-nz7qt"
Sep 17 18:50:30.954571 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:30.954538    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8wdr\" (UniqueName: \"kubernetes.io/projected/2f9542fb-b8fb-42f0-9411-e77a6e51ec36-kube-api-access-t8wdr\") pod \"must-gather-nz7qt\" (UID: \"2f9542fb-b8fb-42f0-9411-e77a6e51ec36\") " pod="openshift-must-gather-rkfh7/must-gather-nz7qt"
Sep 17 18:50:31.003525 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:31.003499    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rkfh7/must-gather-nz7qt"
Sep 17 18:50:31.275782 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:31.275571    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rkfh7/must-gather-nz7qt"]
Sep 17 18:50:31.278403 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:50:31.278374    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f9542fb_b8fb_42f0_9411_e77a6e51ec36.slice/crio-3d52fdf977b7fac2d87cd006974da77808a2cfadb08a480e4089a8b6069bd8ca WatchSource:0}: Error finding container 3d52fdf977b7fac2d87cd006974da77808a2cfadb08a480e4089a8b6069bd8ca: Status 404 returned error can't find the container with id 3d52fdf977b7fac2d87cd006974da77808a2cfadb08a480e4089a8b6069bd8ca
Sep 17 18:50:31.280214 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:31.280192    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:50:32.229191 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:32.229148    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rkfh7/must-gather-nz7qt" event={"ID":"2f9542fb-b8fb-42f0-9411-e77a6e51ec36","Type":"ContainerStarted","Data":"3d52fdf977b7fac2d87cd006974da77808a2cfadb08a480e4089a8b6069bd8ca"}
Sep 17 18:50:38.261812 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:38.261768    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rkfh7/must-gather-nz7qt" event={"ID":"2f9542fb-b8fb-42f0-9411-e77a6e51ec36","Type":"ContainerStarted","Data":"88347b66b40f15f50b01ee2c5f3c317353fa307eb064c0c44eb230521f1fa7d6"}
Sep 17 18:50:38.261812 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:38.261815    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rkfh7/must-gather-nz7qt" event={"ID":"2f9542fb-b8fb-42f0-9411-e77a6e51ec36","Type":"ContainerStarted","Data":"0c6c8e97375e03bb4cd515ebf09b2f525b61050984d7c3cd8c7a2781acfd9d48"}
Sep 17 18:50:38.277299 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:38.277248    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rkfh7/must-gather-nz7qt" podStartSLOduration=2.319580774 podStartE2EDuration="8.277233689s" totalImagesPullingTime="5.957652915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:50:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:50:38.276078045 +0000 UTC m=+4549.367384815" watchObservedRunningTime="2026-09-17 18:50:38.277233689 +0000 UTC m=+4549.368540453"
Sep 17 18:50:55.259616 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:55.259586    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-8j98m_e5c0588a-302e-421f-be89-fe1b306900be/kuadrant-console-plugin/0.log"
Sep 17 18:50:55.353651 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:50:55.353625    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-6lknq_0c053a59-a277-4666-8612-aae4851ddfae/limitador/0.log"
Sep 17 18:51:00.365296 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:00.365256    2583 generic.go:356] "Generic (PLEG): container finished" podID="2f9542fb-b8fb-42f0-9411-e77a6e51ec36" containerID="0c6c8e97375e03bb4cd515ebf09b2f525b61050984d7c3cd8c7a2781acfd9d48" exitCode=0
Sep 17 18:51:00.365783 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:00.365330    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rkfh7/must-gather-nz7qt" event={"ID":"2f9542fb-b8fb-42f0-9411-e77a6e51ec36","Type":"ContainerDied","Data":"0c6c8e97375e03bb4cd515ebf09b2f525b61050984d7c3cd8c7a2781acfd9d48"}
Sep 17 18:51:00.365783 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:00.365651    2583 scope.go:117] "RemoveContainer" containerID="0c6c8e97375e03bb4cd515ebf09b2f525b61050984d7c3cd8c7a2781acfd9d48"
Sep 17 18:51:00.676794 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:00.676712    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-rkfh7_must-gather-nz7qt_2f9542fb-b8fb-42f0-9411-e77a6e51ec36/gather/0.log"
Sep 17 18:51:01.214712 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:01.214677    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fdhq6/must-gather-jjtpm"]
Sep 17 18:51:01.220158 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:01.220138    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fdhq6/must-gather-jjtpm"
Sep 17 18:51:01.222325 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:01.222295    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fdhq6\"/\"default-dockercfg-2qc86\""
Sep 17 18:51:01.222469 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:01.222447    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fdhq6\"/\"kube-root-ca.crt\""
Sep 17 18:51:01.222551 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:01.222460    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fdhq6\"/\"openshift-service-ca.crt\""
Sep 17 18:51:01.229490 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:01.229467    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fdhq6/must-gather-jjtpm"]
Sep 17 18:51:01.300142 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:01.300101    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lhw8\" (UniqueName: \"kubernetes.io/projected/75afdad4-1061-4801-b5e9-c3c9855be537-kube-api-access-7lhw8\") pod \"must-gather-jjtpm\" (UID: \"75afdad4-1061-4801-b5e9-c3c9855be537\") " pod="openshift-must-gather-fdhq6/must-gather-jjtpm"
Sep 17 18:51:01.300142 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:01.300144    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/75afdad4-1061-4801-b5e9-c3c9855be537-must-gather-output\") pod \"must-gather-jjtpm\" (UID: \"75afdad4-1061-4801-b5e9-c3c9855be537\") " pod="openshift-must-gather-fdhq6/must-gather-jjtpm"
Sep 17 18:51:01.400792 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:01.400756    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lhw8\" (UniqueName: \"kubernetes.io/projected/75afdad4-1061-4801-b5e9-c3c9855be537-kube-api-access-7lhw8\") pod \"must-gather-jjtpm\" (UID: \"75afdad4-1061-4801-b5e9-c3c9855be537\") " pod="openshift-must-gather-fdhq6/must-gather-jjtpm"
Sep 17 18:51:01.401294 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:01.400804    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/75afdad4-1061-4801-b5e9-c3c9855be537-must-gather-output\") pod \"must-gather-jjtpm\" (UID: \"75afdad4-1061-4801-b5e9-c3c9855be537\") " pod="openshift-must-gather-fdhq6/must-gather-jjtpm"
Sep 17 18:51:01.401294 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:01.401119    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/75afdad4-1061-4801-b5e9-c3c9855be537-must-gather-output\") pod \"must-gather-jjtpm\" (UID: \"75afdad4-1061-4801-b5e9-c3c9855be537\") " pod="openshift-must-gather-fdhq6/must-gather-jjtpm"
Sep 17 18:51:01.409008 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:01.408983    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lhw8\" (UniqueName: \"kubernetes.io/projected/75afdad4-1061-4801-b5e9-c3c9855be537-kube-api-access-7lhw8\") pod \"must-gather-jjtpm\" (UID: \"75afdad4-1061-4801-b5e9-c3c9855be537\") " pod="openshift-must-gather-fdhq6/must-gather-jjtpm"
Sep 17 18:51:01.529371 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:01.529289    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fdhq6/must-gather-jjtpm"
Sep 17 18:51:01.798072 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:01.798043    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fdhq6/must-gather-jjtpm"]
Sep 17 18:51:01.800498 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:51:01.800467    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75afdad4_1061_4801_b5e9_c3c9855be537.slice/crio-d80aa0629938ba68c071e3364d033a7f11f6868ced87d7c6272e98b519226455 WatchSource:0}: Error finding container d80aa0629938ba68c071e3364d033a7f11f6868ced87d7c6272e98b519226455: Status 404 returned error can't find the container with id d80aa0629938ba68c071e3364d033a7f11f6868ced87d7c6272e98b519226455
Sep 17 18:51:02.374303 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:02.374261    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fdhq6/must-gather-jjtpm" event={"ID":"75afdad4-1061-4801-b5e9-c3c9855be537","Type":"ContainerStarted","Data":"d80aa0629938ba68c071e3364d033a7f11f6868ced87d7c6272e98b519226455"}
Sep 17 18:51:03.383265 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:03.383224    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fdhq6/must-gather-jjtpm" event={"ID":"75afdad4-1061-4801-b5e9-c3c9855be537","Type":"ContainerStarted","Data":"f0abe557ced8fb1f64370477107ac39f07cb108bf1ed44ced0405f95d424ea4f"}
Sep 17 18:51:03.383762 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:03.383282    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fdhq6/must-gather-jjtpm" event={"ID":"75afdad4-1061-4801-b5e9-c3c9855be537","Type":"ContainerStarted","Data":"fd1eb477646d1b0ccaf23598f3eeba84b4fe0fa4b9f006b85c8a27b8a3d6e8be"}
Sep 17 18:51:03.398842 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:03.398785    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fdhq6/must-gather-jjtpm" podStartSLOduration=1.573970498 podStartE2EDuration="2.398766206s" totalImagesPullingTime="824.795708ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:51:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:51:03.398201376 +0000 UTC m=+4574.489508143" watchObservedRunningTime="2026-09-17 18:51:03.398766206 +0000 UTC m=+4574.490072973"
Sep 17 18:51:04.178997 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:04.178959    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-c46w4_4696b495-138f-4b5a-838b-d13ec3d8a3a6/global-pull-secret-syncer/0.log"
Sep 17 18:51:04.303740 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:04.303710    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gtn5h_798a6256-4107-44bb-9da2-b3a7c769eda1/konnectivity-agent/0.log"
Sep 17 18:51:04.388978 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:04.388953    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-30.ec2.internal_710fcb8c6a77db50f998ce1b3ff8fde1/haproxy/0.log"
Sep 17 18:51:06.062463 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:06.062416    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-rkfh7/must-gather-nz7qt"]
Sep 17 18:51:06.062997 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:06.062697    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-rkfh7/must-gather-nz7qt" podUID="2f9542fb-b8fb-42f0-9411-e77a6e51ec36" containerName="copy" containerID="cri-o://88347b66b40f15f50b01ee2c5f3c317353fa307eb064c0c44eb230521f1fa7d6" gracePeriod=2
Sep 17 18:51:06.065032 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:51:06.064995    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-nz7qt\" is forbidden: User \"system:node:ip-10-0-141-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-rkfh7\": no relationship found between node 'ip-10-0-141-30.ec2.internal' and this object" podUID="2f9542fb-b8fb-42f0-9411-e77a6e51ec36" pod="openshift-must-gather-rkfh7/must-gather-nz7qt"
Sep 17 18:51:06.066992 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:06.066947    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-rkfh7/must-gather-nz7qt"]
Sep 17 18:51:06.405108 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:06.404999    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-rkfh7_must-gather-nz7qt_2f9542fb-b8fb-42f0-9411-e77a6e51ec36/copy/0.log"
Sep 17 18:51:06.406580 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:06.406555    2583 generic.go:356] "Generic (PLEG): container finished" podID="2f9542fb-b8fb-42f0-9411-e77a6e51ec36" containerID="88347b66b40f15f50b01ee2c5f3c317353fa307eb064c0c44eb230521f1fa7d6" exitCode=143
Sep 17 18:51:06.445022 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:06.444994    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-rkfh7_must-gather-nz7qt_2f9542fb-b8fb-42f0-9411-e77a6e51ec36/copy/0.log"
Sep 17 18:51:06.447165 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:06.446902    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rkfh7/must-gather-nz7qt"
Sep 17 18:51:06.450682 ip-10-0-141-30 kubenswrapper[2583]: E0917 18:51:06.450284    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-nz7qt\" is forbidden: User \"system:node:ip-10-0-141-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-rkfh7\": no relationship found between node 'ip-10-0-141-30.ec2.internal' and this object" podUID="2f9542fb-b8fb-42f0-9411-e77a6e51ec36" pod="openshift-must-gather-rkfh7/must-gather-nz7qt"
Sep 17 18:51:06.545518 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:06.545475    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8wdr\" (UniqueName: \"kubernetes.io/projected/2f9542fb-b8fb-42f0-9411-e77a6e51ec36-kube-api-access-t8wdr\") pod \"2f9542fb-b8fb-42f0-9411-e77a6e51ec36\" (UID: \"2f9542fb-b8fb-42f0-9411-e77a6e51ec36\") "
Sep 17 18:51:06.545727 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:06.545546    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2f9542fb-b8fb-42f0-9411-e77a6e51ec36-must-gather-output\") pod \"2f9542fb-b8fb-42f0-9411-e77a6e51ec36\" (UID: \"2f9542fb-b8fb-42f0-9411-e77a6e51ec36\") "
Sep 17 18:51:06.550268 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:06.550215    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f9542fb-b8fb-42f0-9411-e77a6e51ec36-kube-api-access-t8wdr" (OuterVolumeSpecName: "kube-api-access-t8wdr") pod "2f9542fb-b8fb-42f0-9411-e77a6e51ec36" (UID: "2f9542fb-b8fb-42f0-9411-e77a6e51ec36"). InnerVolumeSpecName "kube-api-access-t8wdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:51:06.553866 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:06.553834    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9542fb-b8fb-42f0-9411-e77a6e51ec36-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "2f9542fb-b8fb-42f0-9411-e77a6e51ec36" (UID: "2f9542fb-b8fb-42f0-9411-e77a6e51ec36"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:51:06.647029 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:06.646956    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8wdr\" (UniqueName: \"kubernetes.io/projected/2f9542fb-b8fb-42f0-9411-e77a6e51ec36-kube-api-access-t8wdr\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:51:06.647029 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:06.646998    2583 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2f9542fb-b8fb-42f0-9411-e77a6e51ec36-must-gather-output\") on node \"ip-10-0-141-30.ec2.internal\" DevicePath \"\""
Sep 17 18:51:07.043864 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:07.043804    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f9542fb-b8fb-42f0-9411-e77a6e51ec36" path="/var/lib/kubelet/pods/2f9542fb-b8fb-42f0-9411-e77a6e51ec36/volumes"
Sep 17 18:51:07.414913 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:07.414820    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-rkfh7_must-gather-nz7qt_2f9542fb-b8fb-42f0-9411-e77a6e51ec36/copy/0.log"
Sep 17 18:51:07.418410 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:07.415947    2583 scope.go:117] "RemoveContainer" containerID="88347b66b40f15f50b01ee2c5f3c317353fa307eb064c0c44eb230521f1fa7d6"
Sep 17 18:51:07.418410 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:07.416130    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rkfh7/must-gather-nz7qt"
Sep 17 18:51:07.435851 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:07.434045    2583 scope.go:117] "RemoveContainer" containerID="0c6c8e97375e03bb4cd515ebf09b2f525b61050984d7c3cd8c7a2781acfd9d48"
Sep 17 18:51:09.504487 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:09.504457    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-8j98m_e5c0588a-302e-421f-be89-fe1b306900be/kuadrant-console-plugin/0.log"
Sep 17 18:51:09.614676 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:09.614619    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-6lknq_0c053a59-a277-4666-8612-aae4851ddfae/limitador/0.log"
Sep 17 18:51:11.132162 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:11.132121    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-5ftqq_f11e6c11-e031-485d-8ffa-f71b00f75347/cluster-monitoring-operator/0.log"
Sep 17 18:51:11.162870 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:11.162839    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-qn64b_215cb362-1c54-4086-83a8-99c96e59e90d/kube-state-metrics/0.log"
Sep 17 18:51:11.185270 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:11.185243    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-qn64b_215cb362-1c54-4086-83a8-99c96e59e90d/kube-rbac-proxy-main/0.log"
Sep 17 18:51:11.206862 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:11.206830    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-qn64b_215cb362-1c54-4086-83a8-99c96e59e90d/kube-rbac-proxy-self/0.log"
Sep 17 18:51:11.486691 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:11.486562    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t5vx9_031c5c73-85cc-465f-b4c5-ac1b82042ac6/node-exporter/0.log"
Sep 17 18:51:11.514489 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:11.514463    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t5vx9_031c5c73-85cc-465f-b4c5-ac1b82042ac6/kube-rbac-proxy/0.log"
Sep 17 18:51:11.539701 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:11.539673    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t5vx9_031c5c73-85cc-465f-b4c5-ac1b82042ac6/init-textfile/0.log"
Sep 17 18:51:14.022560 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:14.022503    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fdhq6/perf-node-gather-daemonset-d2q72"]
Sep 17 18:51:14.023222 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:14.022959    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f9542fb-b8fb-42f0-9411-e77a6e51ec36" containerName="gather"
Sep 17 18:51:14.023222 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:14.022977    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f9542fb-b8fb-42f0-9411-e77a6e51ec36" containerName="gather"
Sep 17 18:51:14.023222 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:14.023011    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f9542fb-b8fb-42f0-9411-e77a6e51ec36" containerName="copy"
Sep 17 18:51:14.023222 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:14.023021    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f9542fb-b8fb-42f0-9411-e77a6e51ec36" containerName="copy"
Sep 17 18:51:14.023222 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:14.023095    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f9542fb-b8fb-42f0-9411-e77a6e51ec36" containerName="gather"
Sep 17 18:51:14.023222 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:14.023108    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f9542fb-b8fb-42f0-9411-e77a6e51ec36" containerName="copy"
Sep 17 18:51:14.026257 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:14.026224    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-d2q72"
Sep 17 18:51:14.039412 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:14.039386    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fdhq6/perf-node-gather-daemonset-d2q72"]
Sep 17 18:51:14.140867 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:14.140824    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e61d322-e6c8-41a4-8ff1-13cd35a6904a-lib-modules\") pod \"perf-node-gather-daemonset-d2q72\" (UID: \"1e61d322-e6c8-41a4-8ff1-13cd35a6904a\") " pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-d2q72"
Sep 17 18:51:14.141070 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:14.140883    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e61d322-e6c8-41a4-8ff1-13cd35a6904a-sys\") pod \"perf-node-gather-daemonset-d2q72\" (UID: \"1e61d322-e6c8-41a4-8ff1-13cd35a6904a\") " pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-d2q72"
Sep 17 18:51:14.141070 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:14.140910    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1e61d322-e6c8-41a4-8ff1-13cd35a6904a-podres\") pod \"perf-node-gather-daemonset-d2q72\" (UID: \"1e61d322-e6c8-41a4-8ff1-13cd35a6904a\") " pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-d2q72"
Sep 17 18:51:14.141070 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:14.140942    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1e61d322-e6c8-41a4-8ff1-13cd35a6904a-proc\") pod \"perf-node-gather-daemonset-d2q72\" (UID: \"1e61d322-e6c8-41a4-8ff1-13cd35a6904a\") " pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-d2q72"
Sep 17 18:51:14.141070 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:14.140979    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rn9z\" (UniqueName: \"kubernetes.io/projected/1e61d322-e6c8-41a4-8ff1-13cd35a6904a-kube-api-access-9rn9z\") pod \"perf-node-gather-daemonset-d2q72\" (UID: \"1e61d322-e6c8-41a4-8ff1-13cd35a6904a\") " pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-d2q72"
Sep 17 18:51:14.242881 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:14.242296    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e61d322-e6c8-41a4-8ff1-13cd35a6904a-lib-modules\") pod \"perf-node-gather-daemonset-d2q72\" (UID: \"1e61d322-e6c8-41a4-8ff1-13cd35a6904a\") " pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-d2q72"
Sep 17 18:51:14.242881 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:14.242354    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e61d322-e6c8-41a4-8ff1-13cd35a6904a-sys\") pod \"perf-node-gather-daemonset-d2q72\" (UID: \"1e61d322-e6c8-41a4-8ff1-13cd35a6904a\") " pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-d2q72"
Sep 17 18:51:14.242881 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:14.242389    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1e61d322-e6c8-41a4-8ff1-13cd35a6904a-podres\") pod \"perf-node-gather-daemonset-d2q72\" (UID: \"1e61d322-e6c8-41a4-8ff1-13cd35a6904a\") " pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-d2q72"
Sep 17 18:51:14.242881 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:14.242423    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1e61d322-e6c8-41a4-8ff1-13cd35a6904a-proc\") pod \"perf-node-gather-daemonset-d2q72\" (UID: \"1e61d322-e6c8-41a4-8ff1-13cd35a6904a\") " pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-d2q72"
Sep 17 18:51:14.242881 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:14.242466    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rn9z\" (UniqueName: \"kubernetes.io/projected/1e61d322-e6c8-41a4-8ff1-13cd35a6904a-kube-api-access-9rn9z\") pod \"perf-node-gather-daemonset-d2q72\" (UID: \"1e61d322-e6c8-41a4-8ff1-13cd35a6904a\") " pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-d2q72"
Sep 17 18:51:14.242881 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:14.242482    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e61d322-e6c8-41a4-8ff1-13cd35a6904a-lib-modules\") pod \"perf-node-gather-daemonset-d2q72\" (UID: \"1e61d322-e6c8-41a4-8ff1-13cd35a6904a\") " pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-d2q72"
Sep 17 18:51:14.242881 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:14.242676    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1e61d322-e6c8-41a4-8ff1-13cd35a6904a-podres\") pod \"perf-node-gather-daemonset-d2q72\" (UID: \"1e61d322-e6c8-41a4-8ff1-13cd35a6904a\") " pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-d2q72"
Sep 17 18:51:14.242881 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:14.242743    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1e61d322-e6c8-41a4-8ff1-13cd35a6904a-proc\") pod \"perf-node-gather-daemonset-d2q72\" (UID: \"1e61d322-e6c8-41a4-8ff1-13cd35a6904a\") " pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-d2q72"
Sep 17 18:51:14.242881 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:14.242852    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e61d322-e6c8-41a4-8ff1-13cd35a6904a-sys\") pod \"perf-node-gather-daemonset-d2q72\" (UID: \"1e61d322-e6c8-41a4-8ff1-13cd35a6904a\") " pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-d2q72"
Sep 17 18:51:14.253490 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:14.253449    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rn9z\" (UniqueName: \"kubernetes.io/projected/1e61d322-e6c8-41a4-8ff1-13cd35a6904a-kube-api-access-9rn9z\") pod \"perf-node-gather-daemonset-d2q72\" (UID: \"1e61d322-e6c8-41a4-8ff1-13cd35a6904a\") " pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-d2q72"
Sep 17 18:51:14.345743 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:14.345699    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-d2q72"
Sep 17 18:51:14.482953 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:14.482917    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-b5556f557-9nrrc_6aadab25-7d9c-4796-ba49-b1f3c0643282/console/0.log"
Sep 17 18:51:14.629219 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:14.629190    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fdhq6/perf-node-gather-daemonset-d2q72"]
Sep 17 18:51:14.631845 ip-10-0-141-30 kubenswrapper[2583]: W0917 18:51:14.631816    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1e61d322_e6c8_41a4_8ff1_13cd35a6904a.slice/crio-823105358b2815a0245e2eab229ab130f8805a80939d50e278e6fd773a242b58 WatchSource:0}: Error finding container 823105358b2815a0245e2eab229ab130f8805a80939d50e278e6fd773a242b58: Status 404 returned error can't find the container with id 823105358b2815a0245e2eab229ab130f8805a80939d50e278e6fd773a242b58
Sep 17 18:51:15.074221 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:15.074143    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-hx6xd_4893c222-3bba-4071-ae44-3f04bd4c523c/volume-data-source-validator/0.log"
Sep 17 18:51:15.469052 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:15.469016    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-d2q72" event={"ID":"1e61d322-e6c8-41a4-8ff1-13cd35a6904a","Type":"ContainerStarted","Data":"bba6ce5686bd46b689975e4878cb13c2ace6e87d01edd426fb5f58868f67969a"}
Sep 17 18:51:15.469275 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:15.469077    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-d2q72" event={"ID":"1e61d322-e6c8-41a4-8ff1-13cd35a6904a","Type":"ContainerStarted","Data":"823105358b2815a0245e2eab229ab130f8805a80939d50e278e6fd773a242b58"}
Sep 17 18:51:15.469275 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:15.469196    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-d2q72"
Sep 17 18:51:15.488639 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:15.488584    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-d2q72" podStartSLOduration=1.488568759 podStartE2EDuration="1.488568759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:51:15.487074705 +0000 UTC m=+4586.578381492" watchObservedRunningTime="2026-09-17 18:51:15.488568759 +0000 UTC m=+4586.579875520"
Sep 17 18:51:16.085891 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:16.085864    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d6z4r_dad536d7-d27e-4d63-96e9-a6654126532d/dns/0.log"
Sep 17 18:51:16.107266 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:16.107241    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d6z4r_dad536d7-d27e-4d63-96e9-a6654126532d/kube-rbac-proxy/0.log"
Sep 17 18:51:16.283703 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:16.283538    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gzt58_b92cf805-a323-422f-b021-6e8d9bce0099/dns-node-resolver/0.log"
Sep 17 18:51:16.887818 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:16.887792    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ph456_a20d51c6-5fef-4251-9e8f-ee4318f97741/node-ca/0.log"
Sep 17 18:51:18.554528 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:18.554497    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-q5nq9_be8c6ba8-3ad9-4988-b871-01c0d8807b10/serve-healthcheck-canary/0.log"
Sep 17 18:51:19.168431 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:19.168401    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xvk2c_61cae3db-f72a-4441-bc97-46cc985a16b2/kube-rbac-proxy/0.log"
Sep 17 18:51:19.191360 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:19.191333    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xvk2c_61cae3db-f72a-4441-bc97-46cc985a16b2/exporter/0.log"
Sep 17 18:51:19.216650 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:19.216626    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xvk2c_61cae3db-f72a-4441-bc97-46cc985a16b2/extractor/0.log"
Sep 17 18:51:21.485131 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:21.485101    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fdhq6/perf-node-gather-daemonset-d2q72"
Sep 17 18:51:22.158722 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:22.158690    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6796bd7874-c2qsq_e4905595-4329-4cb7-928d-3145b47235ac/manager/0.log"
Sep 17 18:51:23.733896 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:23.733862    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-8v82t_c860729e-690d-45bf-936f-a6c12e01415a/s3-init/0.log"
Sep 17 18:51:23.753054 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:23.753026    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-8v82t_c860729e-690d-45bf-936f-a6c12e01415a/download-hf-model/0.log"
Sep 17 18:51:29.077221 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:29.077189    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-ntb24_38abcd71-ad24-4ef8-b192-6124f30e33fd/kube-storage-version-migrator-operator/1.log"
Sep 17 18:51:29.078088 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:29.078070    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-ntb24_38abcd71-ad24-4ef8-b192-6124f30e33fd/kube-storage-version-migrator-operator/0.log"
Sep 17 18:51:30.073324 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:30.073295    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7rbtw_ebc5fcd3-7a3c-4922-b880-c677bdb77777/kube-multus-additional-cni-plugins/0.log"
Sep 17 18:51:30.095209 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:30.095180    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7rbtw_ebc5fcd3-7a3c-4922-b880-c677bdb77777/egress-router-binary-copy/0.log"
Sep 17 18:51:30.116126 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:30.116106    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7rbtw_ebc5fcd3-7a3c-4922-b880-c677bdb77777/cni-plugins/0.log"
Sep 17 18:51:30.137504 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:30.137480    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7rbtw_ebc5fcd3-7a3c-4922-b880-c677bdb77777/bond-cni-plugin/0.log"
Sep 17 18:51:30.160192 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:30.160162    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7rbtw_ebc5fcd3-7a3c-4922-b880-c677bdb77777/routeoverride-cni/0.log"
Sep 17 18:51:30.183274 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:30.183248    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7rbtw_ebc5fcd3-7a3c-4922-b880-c677bdb77777/whereabouts-cni-bincopy/0.log"
Sep 17 18:51:30.204905 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:30.204878    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7rbtw_ebc5fcd3-7a3c-4922-b880-c677bdb77777/whereabouts-cni/0.log"
Sep 17 18:51:30.592568 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:30.592537    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-gnhzr_3541d447-ae55-428c-9ab4-620f06f9c2fa/kube-multus/0.log"
Sep 17 18:51:30.655210 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:30.655183    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fh98p_ae10190f-448c-4e69-a268-85f77c85d7b6/network-metrics-daemon/0.log"
Sep 17 18:51:30.675491 ip-10-0-141-30 kubenswrapper[2583]: I0917 18:51:30.675452    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fh98p_ae10190f-448c-4e69-a268-85f77c85d7b6/kube-rbac-proxy/0.log"