Sep 08 13:13:48.632630 ip-10-0-128-147 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 08 13:13:48.632641 ip-10-0-128-147 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 08 13:13:48.632650 ip-10-0-128-147 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 08 13:13:48.633016 ip-10-0-128-147 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 08 13:13:58.868801 ip-10-0-128-147 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 08 13:13:58.868815 ip-10-0-128-147 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 9193d6bd429f4baaae30aa4c5b8e5fc9 --
Sep 08 13:15:36.420899 ip-10-0-128-147 systemd[1]: Starting Kubernetes Kubelet...
Sep 08 13:15:36.880001 ip-10-0-128-147 kubenswrapper[2558]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 13:15:36.880001 ip-10-0-128-147 kubenswrapper[2558]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 08 13:15:36.880001 ip-10-0-128-147 kubenswrapper[2558]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 13:15:36.880001 ip-10-0-128-147 kubenswrapper[2558]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 08 13:15:36.880001 ip-10-0-128-147 kubenswrapper[2558]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 13:15:36.881056 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.880951    2558 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 08 13:15:36.886126 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886107    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:15:36.886172 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886128    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:15:36.886172 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886136    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:15:36.886172 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886139    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:15:36.886172 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886143    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:15:36.886172 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886148    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:15:36.886172 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886153    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:15:36.886172 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886157    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:15:36.886172 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886161    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:15:36.886172 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886165    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:15:36.886172 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886168    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:15:36.886172 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886172    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:15:36.886172 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886176    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:15:36.886461 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886180    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:15:36.886461 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886184    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:15:36.886461 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886188    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:15:36.886461 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886192    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:15:36.886461 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886207    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:15:36.886461 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886211    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:15:36.886461 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886217    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:15:36.886461 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886223    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:15:36.886461 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886228    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:15:36.886461 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886232    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:15:36.886461 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886236    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:15:36.886461 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886240    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:15:36.886461 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886245    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:15:36.886461 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886249    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:15:36.886461 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886254    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:15:36.886461 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886259    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:15:36.886461 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886263    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:15:36.886461 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886266    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:15:36.886461 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886270    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:15:36.886461 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886274    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:15:36.886948 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886278    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:15:36.886948 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886282    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:15:36.886948 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886285    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:15:36.886948 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886291    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:15:36.886948 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886296    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:15:36.886948 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886301    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:15:36.886948 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886305    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:15:36.886948 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886309    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:15:36.886948 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886320    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:15:36.886948 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886324    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:15:36.886948 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886328    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:15:36.886948 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886331    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:15:36.886948 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886335    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:15:36.886948 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886340    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:15:36.886948 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886344    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:15:36.886948 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886348    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:15:36.886948 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886352    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:15:36.886948 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886356    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:15:36.886948 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886360    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:15:36.887410 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886364    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:15:36.887410 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886367    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:15:36.887410 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886371    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:15:36.887410 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886375    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:15:36.887410 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886379    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:15:36.887410 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886382    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:15:36.887410 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886386    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:15:36.887410 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886392    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:15:36.887410 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886397    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:15:36.887410 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886401    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:15:36.887410 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886405    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:15:36.887410 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886409    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:15:36.887410 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886412    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:15:36.887410 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886416    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:15:36.887410 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886420    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:15:36.887410 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886424    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:15:36.887410 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886428    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:15:36.887410 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886431    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:15:36.887410 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886435    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:15:36.887888 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886441    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:15:36.887888 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886445    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:15:36.887888 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886448    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:15:36.887888 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886452    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:15:36.887888 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886456    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:15:36.887888 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886460    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:15:36.887888 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886463    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:15:36.887888 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886470    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:15:36.887888 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886474    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:15:36.887888 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886478    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:15:36.887888 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886481    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:15:36.887888 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886487    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:15:36.887888 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886491    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:15:36.887888 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886494    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:15:36.887888 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886498    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:15:36.887888 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886502    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:15:36.887888 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886506    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:15:36.887888 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886510    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:15:36.887888 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886514    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:15:36.887888 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886518    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:15:36.888368 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886521    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:15:36.888368 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886525    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:15:36.888368 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886529    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:15:36.888368 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886533    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:15:36.888368 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886537    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:15:36.888368 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886541    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:15:36.888368 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886545    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:15:36.888368 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.886549    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:15:36.888368 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887533    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:15:36.888368 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887540    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:15:36.888368 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887544    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:15:36.888368 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887549    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:15:36.888368 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887553    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:15:36.888368 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887557    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:15:36.888368 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887561    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:15:36.888368 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887566    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:15:36.888368 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887571    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:15:36.888368 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887577    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:15:36.888368 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887582    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:15:36.888896 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887626    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:15:36.888896 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887634    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:15:36.888896 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887640    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:15:36.888896 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887644    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:15:36.888896 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887650    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:15:36.888896 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887655    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:15:36.888896 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887657    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:15:36.888896 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887660    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:15:36.888896 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887663    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:15:36.888896 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887666    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:15:36.888896 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887669    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:15:36.888896 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887671    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:15:36.888896 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887674    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:15:36.888896 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887677    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:15:36.888896 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887680    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:15:36.888896 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887682    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:15:36.888896 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887685    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:15:36.888896 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887688    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:15:36.888896 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887690    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:15:36.889480 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887693    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:15:36.889480 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887696    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:15:36.889480 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887698    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:15:36.889480 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887701    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:15:36.889480 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887703    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:15:36.889480 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887706    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:15:36.889480 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887709    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:15:36.889480 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887711    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:15:36.889480 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887714    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:15:36.889480 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887716    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:15:36.889480 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887719    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:15:36.889480 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887721    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:15:36.889480 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887725    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:15:36.889480 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887728    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:15:36.889480 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887731    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:15:36.889480 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887734    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:15:36.889480 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887737    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:15:36.889480 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887740    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:15:36.889480 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887744    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:15:36.889967 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887748    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:15:36.889967 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887751    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:15:36.889967 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887754    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:15:36.889967 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887757    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:15:36.889967 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887760    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:15:36.889967 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887763    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:15:36.889967 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887766    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:15:36.889967 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887769    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:15:36.889967 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887771    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:15:36.889967 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887774    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:15:36.889967 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887777    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:15:36.889967 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887780    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:15:36.889967 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887782    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:15:36.889967 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887785    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:15:36.889967 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887788    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:15:36.889967 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887790    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:15:36.889967 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887793    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:15:36.889967 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887796    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:15:36.889967 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887801    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:15:36.889967 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887804    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:15:36.890460 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887806    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:15:36.890460 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887811    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:15:36.890460 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887814    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:15:36.890460 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887817    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:15:36.890460 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887819    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:15:36.890460 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887823    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:15:36.890460 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887826    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:15:36.890460 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887828    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:15:36.890460 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887832    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:15:36.890460 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887835    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:15:36.890460 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887837    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:15:36.890460 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887840    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:15:36.890460 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887842    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:15:36.890460 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887845    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:15:36.890460 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887847    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:15:36.890460 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887850    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:15:36.890460 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887853    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:15:36.890460 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887856    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:15:36.890460 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887858    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:15:36.890460 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887861    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:15:36.890976 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887863    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:15:36.890976 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887866    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:15:36.890976 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887869    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:15:36.890976 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887871    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:15:36.890976 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887874    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:15:36.890976 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887876    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:15:36.890976 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887879    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:15:36.890976 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887882    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:15:36.890976 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887884    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:15:36.890976 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.887887    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:15:36.890976 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888323    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:15:36.890976 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888331    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:15:36.890976 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888335    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:15:36.890976 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888338    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:15:36.890976 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888341    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:15:36.890976 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888345    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:15:36.890976 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888348    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:15:36.890976 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888351    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:15:36.890976 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888354    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:15:36.891436 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888357    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:15:36.891436 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888360    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:15:36.891436 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888363    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:15:36.891436 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888365    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:15:36.891436 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888368    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:15:36.891436 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888370    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:15:36.891436 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888373    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:15:36.891436 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888375    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:15:36.891436 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888378    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:15:36.891436 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888382    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:15:36.891436 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888384    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:15:36.891436 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888387    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:15:36.891436 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888390    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:15:36.891436 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888393    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:15:36.891436 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888396    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:15:36.891436 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888398    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:15:36.891436 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888401    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:15:36.891436 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888403    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:15:36.891436 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888406    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:15:36.891436 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888409    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:15:36.891938 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888411    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:15:36.891938 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888416    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:15:36.891938 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888419    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:15:36.891938 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888423    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:15:36.891938 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888426    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:15:36.891938 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888428    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:15:36.891938 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888431    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:15:36.891938 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888434    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:15:36.891938 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888436    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:15:36.891938 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888439    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:15:36.891938 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888441    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:15:36.891938 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888444    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:15:36.891938 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888447    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:15:36.891938 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888449    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:15:36.891938 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888453    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:15:36.891938 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888458    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:15:36.891938 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888461    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:15:36.891938 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888464    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:15:36.891938 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888467    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:15:36.891938 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888469    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:15:36.892424 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888472    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:15:36.892424 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888474    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:15:36.892424 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888477    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:15:36.892424 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888479    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:15:36.892424 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888482    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:15:36.892424 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888484    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:15:36.892424 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888487    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:15:36.892424 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888489    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:15:36.892424 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888492    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:15:36.892424 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888505    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:15:36.892424 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888508    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:15:36.892424 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888511    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:15:36.892424 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888514    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:15:36.892424 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888518    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:15:36.892424 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888520    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:15:36.892424 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888524    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:15:36.892424 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888527    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:15:36.892424 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888530    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:15:36.892424 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888532    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:15:36.892922 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888535    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:15:36.892922 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888537    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:15:36.892922 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888540    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:15:36.892922 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888543    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:15:36.892922 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888545    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:15:36.892922 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888548    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:15:36.892922 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888551    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:15:36.892922 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888553    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:15:36.892922 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888556    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:15:36.892922 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888559    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:15:36.892922 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888561    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:15:36.892922 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888564    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:15:36.892922 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888567    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:15:36.892922 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888570    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:15:36.892922 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888572    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:15:36.892922 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888575    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:15:36.892922 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888577    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:15:36.892922 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888580    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:15:36.892922 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888582    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:15:36.892922 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888585    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:15:36.893403 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888588    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:15:36.893403 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888606    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:15:36.893403 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888608    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:15:36.893403 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888611    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:15:36.893403 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888613    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:15:36.893403 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888616    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:15:36.893403 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888619    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:15:36.893403 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888622    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:15:36.893403 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888625    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:15:36.893403 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888628    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:15:36.893403 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.888631    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:15:36.893403 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888698    2558 flags.go:64] FLAG: --address="0.0.0.0"
Sep 08 13:15:36.893403 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888743    2558 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 08 13:15:36.893403 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888751    2558 flags.go:64] FLAG: --anonymous-auth="true"
Sep 08 13:15:36.893403 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888756    2558 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 08 13:15:36.893403 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888760    2558 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 08 13:15:36.893403 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888763    2558 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 08 13:15:36.893403 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888768    2558 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 08 13:15:36.893403 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888772    2558 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 08 13:15:36.893403 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888775    2558 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 08 13:15:36.893403 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888778    2558 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 08 13:15:36.893954 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888785    2558 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 08 13:15:36.893954 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888789    2558 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 08 13:15:36.893954 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888792    2558 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 08 13:15:36.893954 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888795    2558 flags.go:64] FLAG: --cgroup-root=""
Sep 08 13:15:36.893954 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888798    2558 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 08 13:15:36.893954 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888801    2558 flags.go:64] FLAG: --client-ca-file=""
Sep 08 13:15:36.893954 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888804    2558 flags.go:64] FLAG: --cloud-provider="external"
Sep 08 13:15:36.893954 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888807    2558 flags.go:64] FLAG: --cluster-dns="[]"
Sep 08 13:15:36.893954 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888812    2558 flags.go:64] FLAG: --cluster-domain=""
Sep 08 13:15:36.893954 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888814    2558 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 08 13:15:36.893954 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888817    2558 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 08 13:15:36.893954 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888821    2558 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 08 13:15:36.893954 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888824    2558 flags.go:64] FLAG: --container-log-max-files="5"
Sep 08 13:15:36.893954 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888828    2558 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 08 13:15:36.893954 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888831    2558 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 08 13:15:36.893954 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888834    2558 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 08 13:15:36.893954 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888837    2558 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 08 13:15:36.893954 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888842    2558 flags.go:64] FLAG: --contention-profiling="false"
Sep 08 13:15:36.893954 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888845    2558 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 08 13:15:36.893954 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888849    2558 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 08 13:15:36.893954 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888853    2558 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 08 13:15:36.893954 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888856    2558 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 08 13:15:36.893954 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888860    2558 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 08 13:15:36.893954 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888863    2558 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 08 13:15:36.894526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888866    2558 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 08 13:15:36.894526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888869    2558 flags.go:64] FLAG: --enable-load-reader="false"
Sep 08 13:15:36.894526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888873    2558 flags.go:64] FLAG: --enable-server="true"
Sep 08 13:15:36.894526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888876    2558 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 08 13:15:36.894526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888881    2558 flags.go:64] FLAG: --event-burst="100"
Sep 08 13:15:36.894526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888883    2558 flags.go:64] FLAG: --event-qps="50"
Sep 08 13:15:36.894526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888886    2558 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 08 13:15:36.894526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888889    2558 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 08 13:15:36.894526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888892    2558 flags.go:64] FLAG: --eviction-hard=""
Sep 08 13:15:36.894526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888897    2558 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 08 13:15:36.894526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888900    2558 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 08 13:15:36.894526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888903    2558 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 08 13:15:36.894526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888906    2558 flags.go:64] FLAG: --eviction-soft=""
Sep 08 13:15:36.894526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888909    2558 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 08 13:15:36.894526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888912    2558 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 08 13:15:36.894526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888915    2558 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 08 13:15:36.894526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888918    2558 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 08 13:15:36.894526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888921    2558 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 08 13:15:36.894526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888924    2558 flags.go:64] FLAG: --fail-swap-on="true"
Sep 08 13:15:36.894526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888927    2558 flags.go:64] FLAG: --feature-gates=""
Sep 08 13:15:36.894526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888931    2558 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 08 13:15:36.894526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888934    2558 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 08 13:15:36.894526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888937    2558 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 08 13:15:36.894526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888940    2558 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 08 13:15:36.894526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888943    2558 flags.go:64] FLAG: --healthz-port="10248"
Sep 08 13:15:36.894526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888947    2558 flags.go:64] FLAG: --help="false"
Sep 08 13:15:36.895202 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888950    2558 flags.go:64] FLAG: --hostname-override="ip-10-0-128-147.ec2.internal"
Sep 08 13:15:36.895202 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888955    2558 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 08 13:15:36.895202 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888958    2558 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 08 13:15:36.895202 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888961    2558 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 08 13:15:36.895202 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888964    2558 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 08 13:15:36.895202 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888967    2558 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 08 13:15:36.895202 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888970    2558 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 08 13:15:36.895202 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888973    2558 flags.go:64] FLAG: --image-service-endpoint=""
Sep 08 13:15:36.895202 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888976    2558 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 08 13:15:36.895202 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888978    2558 flags.go:64] FLAG: --kube-api-burst="100"
Sep 08 13:15:36.895202 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888981    2558 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 08 13:15:36.895202 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888984    2558 flags.go:64] FLAG: --kube-api-qps="50"
Sep 08 13:15:36.895202 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888987    2558 flags.go:64] FLAG: --kube-reserved=""
Sep 08 13:15:36.895202 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888990    2558 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 08 13:15:36.895202 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888993    2558 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 08 13:15:36.895202 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.888996    2558 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 08 13:15:36.895202 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889000    2558 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 08 13:15:36.895202 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889003    2558 flags.go:64] FLAG: --lock-file=""
Sep 08 13:15:36.895202 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889005    2558 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 08 13:15:36.895202 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889008    2558 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 08 13:15:36.895202 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889012    2558 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 08 13:15:36.895202 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889017    2558 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 08 13:15:36.895202 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889020    2558 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 08 13:15:36.895786 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889023    2558 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 08 13:15:36.895786 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889026    2558 flags.go:64] FLAG: --logging-format="text"
Sep 08 13:15:36.895786 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889028    2558 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 08 13:15:36.895786 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889032    2558 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 08 13:15:36.895786 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889034    2558 flags.go:64] FLAG: --manifest-url=""
Sep 08 13:15:36.895786 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889037    2558 flags.go:64] FLAG: --manifest-url-header=""
Sep 08 13:15:36.895786 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889041    2558 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 08 13:15:36.895786 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889044    2558 flags.go:64] FLAG: --max-open-files="1000000"
Sep 08 13:15:36.895786 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889049    2558 flags.go:64] FLAG: --max-pods="110"
Sep 08 13:15:36.895786 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889052    2558 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 08 13:15:36.895786 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889057    2558 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 08 13:15:36.895786 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889060    2558 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 08 13:15:36.895786 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889063    2558 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 08 13:15:36.895786 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889066    2558 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 08 13:15:36.895786 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889069    2558 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 08 13:15:36.895786 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889072    2558 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 08 13:15:36.895786 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889079    2558 flags.go:64] FLAG: --node-status-max-images="50"
Sep 08 13:15:36.895786 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889082    2558 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 08 13:15:36.895786 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889085    2558 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 08 13:15:36.895786 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889088    2558 flags.go:64] FLAG: --pod-cidr=""
Sep 08 13:15:36.895786 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889091    2558 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 08 13:15:36.895786 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889098    2558 flags.go:64] FLAG: --pod-manifest-path=""
Sep 08 13:15:36.895786 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889101    2558 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 08 13:15:36.895786 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889104    2558 flags.go:64] FLAG: --pods-per-core="0"
Sep 08 13:15:36.896389 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889107    2558 flags.go:64] FLAG: --port="10250"
Sep 08 13:15:36.896389 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889110    2558 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 08 13:15:36.896389 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889113    2558 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-002a870d65d65be78"
Sep 08 13:15:36.896389 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889116    2558 flags.go:64] FLAG: --qos-reserved=""
Sep 08 13:15:36.896389 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889119    2558 flags.go:64] FLAG: --read-only-port="10255"
Sep 08 13:15:36.896389 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889122    2558 flags.go:64] FLAG: --register-node="true"
Sep 08 13:15:36.896389 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889125    2558 flags.go:64] FLAG: --register-with-taints=""
Sep 08 13:15:36.896389 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889129    2558 flags.go:64] FLAG: --registry-burst="10"
Sep 08 13:15:36.896389 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889132    2558 flags.go:64] FLAG: --registry-qps="5"
Sep 08 13:15:36.896389 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889135    2558 flags.go:64] FLAG: --reserved-cpus=""
Sep 08 13:15:36.896389 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889137    2558 flags.go:64] FLAG: --reserved-memory=""
Sep 08 13:15:36.896389 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889142    2558 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 08 13:15:36.896389 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889145    2558 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 08 13:15:36.896389 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889148    2558 flags.go:64] FLAG: --rotate-certificates="false"
Sep 08 13:15:36.896389 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889151    2558 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 08 13:15:36.896389 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889154    2558 flags.go:64] FLAG: --runonce="false"
Sep 08 13:15:36.896389 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889158    2558 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 08 13:15:36.896389 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889162    2558 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 08 13:15:36.896389 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889165    2558 flags.go:64] FLAG: --seccomp-default="false"
Sep 08 13:15:36.896389 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889170    2558 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 08 13:15:36.896389 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889173    2558 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 08 13:15:36.896389 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889176    2558 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 08 13:15:36.896389 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889179    2558 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 08 13:15:36.896389 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889182    2558 flags.go:64] FLAG: --storage-driver-password="root"
Sep 08 13:15:36.896389 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889185    2558 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 08 13:15:36.896389 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889188    2558 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 08 13:15:36.897105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889190    2558 flags.go:64] FLAG: --storage-driver-user="root"
Sep 08 13:15:36.897105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889193    2558 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 08 13:15:36.897105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889196    2558 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 08 13:15:36.897105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889199    2558 flags.go:64] FLAG: --system-cgroups=""
Sep 08 13:15:36.897105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889202    2558 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 08 13:15:36.897105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889207    2558 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 08 13:15:36.897105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889210    2558 flags.go:64] FLAG: --tls-cert-file=""
Sep 08 13:15:36.897105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889213    2558 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 08 13:15:36.897105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889217    2558 flags.go:64] FLAG: --tls-min-version=""
Sep 08 13:15:36.897105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889221    2558 flags.go:64] FLAG: --tls-private-key-file=""
Sep 08 13:15:36.897105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889224    2558 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 08 13:15:36.897105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889227    2558 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 08 13:15:36.897105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889229    2558 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 08 13:15:36.897105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889232    2558 flags.go:64] FLAG: --v="2"
Sep 08 13:15:36.897105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889237    2558 flags.go:64] FLAG: --version="false"
Sep 08 13:15:36.897105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889241    2558 flags.go:64] FLAG: --vmodule=""
Sep 08 13:15:36.897105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889244    2558 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 08 13:15:36.897105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.889248    2558 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 08 13:15:36.897105 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889363    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:15:36.897105 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889366    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:15:36.897105 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889369    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:15:36.897105 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889372    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:15:36.897105 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889376    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:15:36.897105 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889379    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:15:36.897711 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889382    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:15:36.897711 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889386    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:15:36.897711 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889388    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:15:36.897711 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889391    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:15:36.897711 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889394    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:15:36.897711 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889397    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:15:36.897711 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889399    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:15:36.897711 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889402    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:15:36.897711 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889405    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:15:36.897711 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889408    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:15:36.897711 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889411    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:15:36.897711 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889413    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:15:36.897711 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889416    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:15:36.897711 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889418    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:15:36.897711 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889421    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:15:36.897711 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889424    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:15:36.897711 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889427    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:15:36.897711 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889429    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:15:36.897711 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889432    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:15:36.897711 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889435    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:15:36.898208 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889438    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:15:36.898208 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889441    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:15:36.898208 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889443    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:15:36.898208 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889446    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:15:36.898208 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889448    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:15:36.898208 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889452    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:15:36.898208 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889456    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:15:36.898208 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889459    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:15:36.898208 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889462    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:15:36.898208 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889465    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:15:36.898208 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889469    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:15:36.898208 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889471    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:15:36.898208 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889474    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:15:36.898208 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889478    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:15:36.898208 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889481    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:15:36.898208 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889484    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:15:36.898208 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889487    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:15:36.898208 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889490    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:15:36.898208 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889492    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:15:36.898685 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889495    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:15:36.898685 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889498    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:15:36.898685 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889501    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:15:36.898685 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889503    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:15:36.898685 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889506    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:15:36.898685 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889508    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:15:36.898685 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889511    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:15:36.898685 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889514    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:15:36.898685 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889517    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:15:36.898685 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889521    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:15:36.898685 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889523    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:15:36.898685 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889526    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:15:36.898685 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889529    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:15:36.898685 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889532    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:15:36.898685 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889534    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:15:36.898685 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889537    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:15:36.898685 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889539    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:15:36.898685 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889542    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:15:36.898685 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889546    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:15:36.898685 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889549    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:15:36.899183 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889552    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:15:36.899183 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889554    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:15:36.899183 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889557    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:15:36.899183 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889561    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:15:36.899183 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889564    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:15:36.899183 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889568    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:15:36.899183 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889571    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:15:36.899183 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889574    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:15:36.899183 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889577    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:15:36.899183 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889580    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:15:36.899183 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889583    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:15:36.899183 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889585    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:15:36.899183 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889588    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:15:36.899183 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889614    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:15:36.899183 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889616    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:15:36.899183 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889619    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:15:36.899183 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889622    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:15:36.899183 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889624    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:15:36.899183 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889627    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:15:36.899183 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889629    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:15:36.899702 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889632    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:15:36.899702 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889634    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:15:36.899702 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889637    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:15:36.899702 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889639    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:15:36.899702 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889642    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:15:36.899702 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889645    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:15:36.899702 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889647    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:15:36.899702 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889650    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:15:36.899702 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889652    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:15:36.899702 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889655    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:15:36.899702 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889658    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:15:36.899702 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889660    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:15:36.899702 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889663    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:15:36.899702 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.889665    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:15:36.899702 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.890415    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 13:15:36.900073 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.899102    2558 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 08 13:15:36.900073 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.899121    2558 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 08 13:15:36.900073 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.899138    2558 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 08 13:15:36.900073 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.899145    2558 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 08 13:15:36.900073 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899217    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:15:36.900073 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899224    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:15:36.900073 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899229    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:15:36.900073 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899233    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:15:36.900073 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899237    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:15:36.900073 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899241    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:15:36.900073 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899246    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:15:36.900073 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899252    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:15:36.900073 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899257    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:15:36.900073 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899262    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:15:36.900073 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899266    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:15:36.900073 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899270    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:15:36.900073 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899274    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:15:36.900073 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899278    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:15:36.900073 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899282    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:15:36.900073 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899286    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:15:36.900563 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899290    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:15:36.900563 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899294    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:15:36.900563 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899298    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:15:36.900563 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899302    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:15:36.900563 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899307    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:15:36.900563 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899311    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:15:36.900563 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899314    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:15:36.900563 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899319    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:15:36.900563 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899324    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:15:36.900563 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899327    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:15:36.900563 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899331    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:15:36.900563 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899334    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:15:36.900563 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899337    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:15:36.900563 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899342    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:15:36.900563 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899347    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:15:36.900563 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899350    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:15:36.900563 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899354    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:15:36.900563 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899358    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:15:36.900563 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899361    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:15:36.901103 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899366    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:15:36.901103 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899369    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:15:36.901103 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899373    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:15:36.901103 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899376    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:15:36.901103 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899380    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:15:36.901103 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899384    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:15:36.901103 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899388    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:15:36.901103 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899392    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:15:36.901103 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899395    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:15:36.901103 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899399    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:15:36.901103 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899402    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:15:36.901103 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899406    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:15:36.901103 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899409    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:15:36.901103 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899413    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:15:36.901103 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899417    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:15:36.901103 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899421    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:15:36.901103 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899425    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:15:36.901103 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899428    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:15:36.901103 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899432    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:15:36.901103 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899436    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:15:36.901831 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899439    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:15:36.901831 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899444    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:15:36.901831 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899448    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:15:36.901831 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899451    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:15:36.901831 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899455    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:15:36.901831 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899459    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:15:36.901831 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899463    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:15:36.901831 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899467    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:15:36.901831 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899472    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:15:36.901831 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899477    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:15:36.901831 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899483    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:15:36.901831 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899489    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:15:36.901831 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899493    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:15:36.901831 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899498    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:15:36.901831 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899501    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:15:36.901831 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899505    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:15:36.901831 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899509    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:15:36.901831 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899512    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:15:36.901831 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899517    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:15:36.902526 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899521    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:15:36.902526 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899524    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:15:36.902526 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899528    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:15:36.902526 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899532    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:15:36.902526 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899536    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:15:36.902526 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899540    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:15:36.902526 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899543    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:15:36.902526 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899546    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:15:36.902526 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899550    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:15:36.902526 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899554    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:15:36.902526 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899558    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:15:36.902526 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899561    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:15:36.902526 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899565    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:15:36.902526 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899568    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:15:36.902526 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899572    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:15:36.902526 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899576    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:15:36.902526 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899580    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:15:36.902526 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899583    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:15:36.902526 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899587    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:15:36.902526 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899605    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:15:36.903034 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899610    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:15:36.903034 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899614    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:15:36.903034 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899617    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:15:36.903034 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899621    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:15:36.903034 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899625    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:15:36.903034 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.899632    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 13:15:36.903034 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899783    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:15:36.903034 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899790    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:15:36.903034 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899794    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:15:36.903034 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899798    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:15:36.903034 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899802    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:15:36.903034 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899806    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:15:36.903034 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899809    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:15:36.903034 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899814    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:15:36.903034 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899820    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:15:36.903415 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899823    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:15:36.903415 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899827    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:15:36.903415 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899831    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:15:36.903415 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899834    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:15:36.903415 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899838    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:15:36.903415 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899842    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:15:36.903415 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899846    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:15:36.903415 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899849    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:15:36.903415 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899852    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:15:36.903415 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899856    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:15:36.903415 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899860    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:15:36.903415 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899864    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:15:36.903415 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899867    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:15:36.903415 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899871    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:15:36.903415 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899875    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:15:36.903415 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899878    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:15:36.903415 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899883    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:15:36.903415 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899888    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:15:36.903415 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899892    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:15:36.903415 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899896    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:15:36.903921 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899899    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:15:36.903921 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899904    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:15:36.903921 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899909    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:15:36.903921 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899914    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:15:36.903921 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899917    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:15:36.903921 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899921    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:15:36.903921 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899925    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:15:36.903921 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899928    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:15:36.903921 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899932    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:15:36.903921 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899936    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:15:36.903921 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899939    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:15:36.903921 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899943    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:15:36.903921 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899946    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:15:36.903921 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899949    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:15:36.903921 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899953    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:15:36.903921 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899957    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:15:36.903921 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899960    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:15:36.903921 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899964    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:15:36.903921 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899967    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:15:36.904454 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899971    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:15:36.904454 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899974    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:15:36.904454 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899979    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:15:36.904454 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899982    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:15:36.904454 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899986    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:15:36.904454 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899989    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:15:36.904454 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899993    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:15:36.904454 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.899997    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:15:36.904454 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900001    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:15:36.904454 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900005    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:15:36.904454 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900009    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:15:36.904454 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900012    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:15:36.904454 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900017    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:15:36.904454 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900020    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:15:36.904454 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900023    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:15:36.904454 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900027    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:15:36.904454 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900031    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:15:36.904454 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900035    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:15:36.904454 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900038    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:15:36.905089 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900043    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:15:36.905089 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900048    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:15:36.905089 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900052    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:15:36.905089 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900056    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:15:36.905089 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900059    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:15:36.905089 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900063    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:15:36.905089 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900067    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:15:36.905089 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900071    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:15:36.905089 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900076    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:15:36.905089 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900080    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:15:36.905089 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900084    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:15:36.905089 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900087    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:15:36.905089 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900090    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:15:36.905089 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900094    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:15:36.905089 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900098    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:15:36.905089 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900102    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:15:36.905089 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900105    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:15:36.905089 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900108    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:15:36.905089 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900112    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:15:36.905551 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900115    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:15:36.905551 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900119    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:15:36.905551 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900124    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:15:36.905551 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900128    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:15:36.905551 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900131    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:15:36.905551 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900134    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:15:36.905551 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900138    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:15:36.905551 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900142    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:15:36.905551 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900146    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:15:36.905551 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900149    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:15:36.905551 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900152    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:15:36.905551 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900156    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:15:36.905551 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:36.900159    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:15:36.905551 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.900167    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 13:15:36.905551 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.900312    2558 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 08 13:15:36.905986 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.905967    2558 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 08 13:15:36.907088 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.907075    2558 server.go:1019] "Starting client certificate rotation"
Sep 08 13:15:36.907207 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.907187    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 13:15:36.907262 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.907232    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 13:15:36.909897 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.909880    2558 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 13:15:36.911089 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.911068    2558 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 13:15:36.920623 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.920606    2558 log.go:25] "Validated CRI v1 runtime API"
Sep 08 13:15:36.924669 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.924653    2558 log.go:25] "Validated CRI v1 image API"
Sep 08 13:15:36.927935 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.927920    2558 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 08 13:15:36.932370 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.932345    2558 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 bf008c7b-8c81-4b39-99df-b866e928864d:/dev/nvme0n1p3 e0462f34-12af-44d3-af8e-101f325dcf0b:/dev/nvme0n1p4]
Sep 08 13:15:36.932438 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.932369    2558 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 08 13:15:36.935780 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.935762    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 13:15:36.939301 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.939192    2558 manager.go:216] Machine: {Timestamp:2026-09-08 13:15:36.937741601 +0000 UTC m=+0.389502527 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101047 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2da60b113056eabd8d6b90cadd5db9 SystemUUID:ec2da60b-1130-56ea-bd8d-6b90cadd5db9 BootID:9193d6bd-429f-4baa-ae30-aa4c5b8e5fc9 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:87:aa:6e:cd:3f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:87:aa:6e:cd:3f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:7a:96:fe:39:98 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 08 13:15:36.939301 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.939297    2558 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 08 13:15:36.939433 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.939421    2558 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 08 13:15:36.943113 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.943084    2558 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 08 13:15:36.943299 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.943119    2558 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-147.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 08 13:15:36.943995 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.943984    2558 topology_manager.go:138] "Creating topology manager with none policy"
Sep 08 13:15:36.944027 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.944001    2558 container_manager_linux.go:309] "Creating device plugin manager"
Sep 08 13:15:36.944027 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.944012    2558 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 13:15:36.944095 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.944031    2558 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 13:15:36.946053 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.946042    2558 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 08 13:15:36.950895 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.950882    2558 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 13:15:36.951033 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.951023    2558 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 08 13:15:36.951065 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.951061    2558 kubelet.go:486] "Attempting to sync node with API server"
Sep 08 13:15:36.951092 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.951074    2558 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 08 13:15:36.951119 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.951091    2558 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 08 13:15:36.951119 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.951106    2558 kubelet.go:398] "Adding apiserver pod source"
Sep 08 13:15:36.951245 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.951127    2558 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 08 13:15:36.952385 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.952358    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 13:15:36.952473 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.952394    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 13:15:36.955062 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.955039    2558 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 08 13:15:36.957880 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.957861    2558 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 08 13:15:36.957969 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.957901    2558 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 08 13:15:36.958090 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.958080    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 08 13:15:36.958125 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.958096    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 08 13:15:36.958125 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.958103    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 08 13:15:36.958125 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.958111    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 08 13:15:36.958125 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.958123    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 08 13:15:36.958278 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.958131    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 08 13:15:36.958752 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.958739    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 08 13:15:36.958795 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.958753    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 08 13:15:36.958795 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.958764    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 08 13:15:36.958795 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.958772    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 08 13:15:36.958795 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.958785    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 08 13:15:36.958795 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.958795    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 08 13:15:36.959627 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.959617    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 08 13:15:36.959627 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.959626    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 08 13:15:36.964218 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.964068    2558 server.go:1268] "Started kubelet"
Sep 08 13:15:36.964402 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.964347    2558 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 08 13:15:36.964402 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.964398    2558 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-147.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 08 13:15:36.964543 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.964469    2558 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 08 13:15:36.964543 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.964469    2558 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 08 13:15:36.965043 ip-10-0-128-147 systemd[1]: Started Kubernetes Kubelet.
Sep 08 13:15:36.966050 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.966028    2558 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 08 13:15:36.967315 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.967294    2558 server.go:310] "Adding debug handlers to kubelet server"
Sep 08 13:15:36.969102 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:36.969072    2558 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 08 13:15:36.970680 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:36.970658    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-147.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 08 13:15:36.970767 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:36.970682    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 08 13:15:36.971095 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:36.969977    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-147.ec2.internal.18d35ad275e8d649  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-147.ec2.internal,UID:ip-10-0-128-147.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-147.ec2.internal,},FirstTimestamp:2026-09-08 13:15:36.964044361 +0000 UTC m=+0.415805284,LastTimestamp:2026-09-08 13:15:36.964044361 +0000 UTC m=+0.415805284,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-147.ec2.internal,}"
Sep 08 13:15:36.972076 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.972058    2558 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 08 13:15:36.972147 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.972079    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 08 13:15:36.972208 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:36.972165    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-147.ec2.internal\" not found"
Sep 08 13:15:36.972828 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.972813    2558 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 08 13:15:36.972934 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.972814    2558 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 08 13:15:36.972989 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.972941    2558 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 08 13:15:36.973180 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.973166    2558 reconstruct.go:97] "Volume reconstruction finished"
Sep 08 13:15:36.973305 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.973289    2558 reconciler.go:29] "Reconciler: start to sync state"
Sep 08 13:15:36.973305 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.973263    2558 factory.go:55] Registering systemd factory
Sep 08 13:15:36.973483 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.973321    2558 factory.go:223] Registration of the systemd container factory successfully
Sep 08 13:15:36.973587 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.973572    2558 factory.go:153] Registering CRI-O factory
Sep 08 13:15:36.973674 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.973608    2558 factory.go:223] Registration of the crio container factory successfully
Sep 08 13:15:36.973674 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.973660    2558 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 08 13:15:36.973733 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.973678    2558 factory.go:103] Registering Raw factory
Sep 08 13:15:36.973733 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.973688    2558 manager.go:1196] Started watching for new ooms in manager
Sep 08 13:15:36.974210 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.974194    2558 manager.go:349] Starting recovery of all containers
Sep 08 13:15:36.974281 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:36.974229    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 08 13:15:36.974346 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:36.974325    2558 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-147.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 08 13:15:36.985770 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.985750    2558 manager.go:354] Recovery completed
Sep 08 13:15:36.990172 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.990158    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:15:36.991847 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.991825    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-147.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:15:36.991908 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.991861    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-147.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:15:36.991908 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.991875    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-147.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:15:36.992453 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.992437    2558 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 08 13:15:36.992453 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.992450    2558 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 08 13:15:36.992601 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.992455    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:15:36.992601 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.992465    2558 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 13:15:36.993201 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.993185    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-147.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:15:36.993251 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.993217    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-147.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:15:36.993251 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.993231    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-147.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:15:36.993892 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:36.993824    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-147.ec2.internal.18d35ad277911173  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-147.ec2.internal,UID:ip-10-0-128-147.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-147.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-147.ec2.internal,},FirstTimestamp:2026-09-08 13:15:36.991846771 +0000 UTC m=+0.443607694,LastTimestamp:2026-09-08 13:15:36.991846771 +0000 UTC m=+0.443607694,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-147.ec2.internal,}"
Sep 08 13:15:36.994759 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.994747    2558 policy_none.go:49] "None policy: Start"
Sep 08 13:15:36.994813 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.994764    2558 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 08 13:15:36.994813 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.994775    2558 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 08 13:15:36.995482 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.995465    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hp6kn"
Sep 08 13:15:36.996758 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:36.996747    2558 policy_none.go:47] "Start"
Sep 08 13:15:37.002390 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.002375    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hp6kn"
Sep 08 13:15:37.003751 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:37.003687    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-147.ec2.internal.18d35ad27791634c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-147.ec2.internal,UID:ip-10-0-128-147.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-147.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-147.ec2.internal,},FirstTimestamp:2026-09-08 13:15:36.991867724 +0000 UTC m=+0.443628648,LastTimestamp:2026-09-08 13:15:36.991867724 +0000 UTC m=+0.443628648,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-147.ec2.internal,}"
Sep 08 13:15:37.041293 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.041276    2558 manager.go:337] "Starting Device Plugin manager"
Sep 08 13:15:37.056195 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:37.041320    2558 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 08 13:15:37.056195 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.041333    2558 server.go:87] "Starting device plugin registration server"
Sep 08 13:15:37.056195 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.041644    2558 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 08 13:15:37.056195 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.041655    2558 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 08 13:15:37.056195 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.041750    2558 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 08 13:15:37.056195 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.041823    2558 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 08 13:15:37.056195 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.041831    2558 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 08 13:15:37.056195 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:37.042391    2558 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 08 13:15:37.056195 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:37.042439    2558 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-147.ec2.internal\" not found"
Sep 08 13:15:37.056195 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.045571    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 08 13:15:37.056195 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.046754    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 08 13:15:37.056195 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.046800    2558 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 08 13:15:37.056195 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.046821    2558 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 08 13:15:37.056195 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:37.046859    2558 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 08 13:15:37.056195 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.049297    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:15:37.142030 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.141932    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:15:37.142970 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.142953    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-147.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:15:37.143060 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.142985    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-147.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:15:37.143060 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.143019    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-147.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:15:37.143060 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.143041    2558 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-147.ec2.internal"
Sep 08 13:15:37.147137 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.147113    2558 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-147.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-147.ec2.internal"]
Sep 08 13:15:37.147209 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.147195    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:15:37.148756 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.148741    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-147.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:15:37.148850 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.148771    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-147.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:15:37.148850 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.148785    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-147.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:15:37.151188 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.151173    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:15:37.151326 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.151289    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-147.ec2.internal"
Sep 08 13:15:37.151382 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.151340    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:15:37.151900 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.151881    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-147.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:15:37.151983 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.151904    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-147.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:15:37.151983 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.151914    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-147.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:15:37.151983 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.151959    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-147.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:15:37.151983 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.151981    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-147.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:15:37.152217 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.152000    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-147.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:15:37.153414 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.153401    2558 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-147.ec2.internal"
Sep 08 13:15:37.153452 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:37.153418    2558 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-147.ec2.internal\": node \"ip-10-0-128-147.ec2.internal\" not found"
Sep 08 13:15:37.154288 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.154274    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-147.ec2.internal"
Sep 08 13:15:37.154362 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.154302    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:15:37.155041 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.155021    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-147.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:15:37.155132 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.155049    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-147.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:15:37.155132 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.155063    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-147.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:15:37.173789 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.173754    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/485a6b3955bdaff3eb439536885dfd93-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-147.ec2.internal\" (UID: \"485a6b3955bdaff3eb439536885dfd93\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-147.ec2.internal"
Sep 08 13:15:37.173789 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.173793    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/485a6b3955bdaff3eb439536885dfd93-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-147.ec2.internal\" (UID: \"485a6b3955bdaff3eb439536885dfd93\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-147.ec2.internal"
Sep 08 13:15:37.173962 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.173812    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/21e691d07f9ad0e45bf40e33342a7334-config\") pod \"kube-apiserver-proxy-ip-10-0-128-147.ec2.internal\" (UID: \"21e691d07f9ad0e45bf40e33342a7334\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-147.ec2.internal"
Sep 08 13:15:37.178677 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:37.178655    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-147.ec2.internal\" not found"
Sep 08 13:15:37.184424 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:37.184406    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-147.ec2.internal\" not found" node="ip-10-0-128-147.ec2.internal"
Sep 08 13:15:37.188815 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:37.188796    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-147.ec2.internal\" not found" node="ip-10-0-128-147.ec2.internal"
Sep 08 13:15:37.274759 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.274726    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/485a6b3955bdaff3eb439536885dfd93-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-147.ec2.internal\" (UID: \"485a6b3955bdaff3eb439536885dfd93\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-147.ec2.internal"
Sep 08 13:15:37.274759 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.274763    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/485a6b3955bdaff3eb439536885dfd93-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-147.ec2.internal\" (UID: \"485a6b3955bdaff3eb439536885dfd93\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-147.ec2.internal"
Sep 08 13:15:37.274955 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.274782    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/21e691d07f9ad0e45bf40e33342a7334-config\") pod \"kube-apiserver-proxy-ip-10-0-128-147.ec2.internal\" (UID: \"21e691d07f9ad0e45bf40e33342a7334\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-147.ec2.internal"
Sep 08 13:15:37.274955 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.274817    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/21e691d07f9ad0e45bf40e33342a7334-config\") pod \"kube-apiserver-proxy-ip-10-0-128-147.ec2.internal\" (UID: \"21e691d07f9ad0e45bf40e33342a7334\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-147.ec2.internal"
Sep 08 13:15:37.274955 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.274841    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/485a6b3955bdaff3eb439536885dfd93-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-147.ec2.internal\" (UID: \"485a6b3955bdaff3eb439536885dfd93\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-147.ec2.internal"
Sep 08 13:15:37.274955 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.274882    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/485a6b3955bdaff3eb439536885dfd93-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-147.ec2.internal\" (UID: \"485a6b3955bdaff3eb439536885dfd93\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-147.ec2.internal"
Sep 08 13:15:37.278861 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:37.278833    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-147.ec2.internal\" not found"
Sep 08 13:15:37.379162 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:37.379129    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-147.ec2.internal\" not found"
Sep 08 13:15:37.479693 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:37.479658    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-147.ec2.internal\" not found"
Sep 08 13:15:37.489602 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.489566    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-147.ec2.internal"
Sep 08 13:15:37.492999 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.492982    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-147.ec2.internal"
Sep 08 13:15:37.580840 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:37.580803    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-147.ec2.internal\" not found"
Sep 08 13:15:37.681270 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:37.681243    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-147.ec2.internal\" not found"
Sep 08 13:15:37.781697 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:37.781628    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-147.ec2.internal\" not found"
Sep 08 13:15:37.861540 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.861500    2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:15:37.881749 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:37.881717    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-147.ec2.internal\" not found"
Sep 08 13:15:37.907327 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.907293    2558 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 08 13:15:37.907495 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.907467    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 13:15:37.907495 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:37.907476    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 13:15:37.982043 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:37.981836    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-147.ec2.internal\" not found"
Sep 08 13:15:38.005770 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.005725    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 13:10:37 +0000 UTC" deadline="2028-03-27 14:59:31.753878323 +0000 UTC"
Sep 08 13:15:38.005770 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.005763    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13585h43m53.748118592s"
Sep 08 13:15:38.082387 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:38.082306    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-147.ec2.internal\" not found"
Sep 08 13:15:38.092780 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.092758    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:15:38.095298 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.095280    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-147.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:15:38.095379 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.095314    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-147.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:15:38.095379 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.095325    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-147.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:15:38.125946 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.125924    2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:15:38.142964 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:38.142911    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod485a6b3955bdaff3eb439536885dfd93.slice/crio-1373d61b6922008b9102ef8450aaa687254a496643400c37118b4e620a4a592b WatchSource:0}: Error finding container 1373d61b6922008b9102ef8450aaa687254a496643400c37118b4e620a4a592b: Status 404 returned error can't find the container with id 1373d61b6922008b9102ef8450aaa687254a496643400c37118b4e620a4a592b
Sep 08 13:15:38.148451 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.148434    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:15:38.172671 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.172642    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-147.ec2.internal"
Sep 08 13:15:38.183901 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.183878    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 13:15:38.185121 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.185105    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-147.ec2.internal"
Sep 08 13:15:38.205247 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.205227    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 13:15:38.541703 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.541667    2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:15:38.736721 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.736689    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:15:38.952842 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.952802    2558 apiserver.go:52] "Watching apiserver"
Sep 08 13:15:38.962350 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.962320    2558 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 08 13:15:38.962840 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.962791    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-br278","kube-system/kube-apiserver-proxy-ip-10-0-128-147.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-147.ec2.internal","openshift-multus/multus-additional-cni-plugins-jtgss","openshift-multus/network-metrics-daemon-q4m28","openshift-network-operator/iptables-alerter-75fqq","openshift-ovn-kubernetes/ovnkube-node-tzwrm","kube-system/global-pull-secret-syncer-g5jt7","kube-system/konnectivity-agent-k9f94","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5zws","openshift-cluster-node-tuning-operator/tuned-k7cdm","openshift-dns/node-resolver-vwvwg","openshift-image-registry/node-ca-7t44n","openshift-multus/multus-zss2s"]
Sep 08 13:15:38.965601 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.965562    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:38.967983 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.967958    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7t44n"
Sep 08 13:15:38.968190 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.968167    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 08 13:15:38.968544 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.968520    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 08 13:15:38.968659 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.968571    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 08 13:15:38.968998 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.968921    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wgq8q\""
Sep 08 13:15:38.968998 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.968934    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 08 13:15:38.968998 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.968928    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 08 13:15:38.968998 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.968945    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 08 13:15:38.970192 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.970170    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 08 13:15:38.970547 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.970515    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 08 13:15:38.970547 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.970531    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 08 13:15:38.970797 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.970563    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zss2s"
Sep 08 13:15:38.970980 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.970955    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9db9d\""
Sep 08 13:15:38.972585 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.972564    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 08 13:15:38.972696 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.972636    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-g99sc\""
Sep 08 13:15:38.972696 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.972646    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 08 13:15:38.973344 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.973321    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jtgss"
Sep 08 13:15:38.974544 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.974524    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 08 13:15:38.974954 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.974836    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 08 13:15:38.975296 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.975278    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 08 13:15:38.976329 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.976308    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 08 13:15:38.976774 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.976752    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 08 13:15:38.976990 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.976972    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pbcsr\""
Sep 08 13:15:38.978330 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.978306    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:15:38.978440 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:38.978400    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4m28" podUID="90dadd3f-c403-4d64-af3a-45b4c68163e5"
Sep 08 13:15:38.980924 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.980903    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-75fqq"
Sep 08 13:15:38.983287 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.983265    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 08 13:15:38.983381 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.983363    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zf4jc\""
Sep 08 13:15:38.983459 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.983271    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 08 13:15:38.983516 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.983461    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:15:38.983590 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.983571    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k9f94"
Sep 08 13:15:38.983996 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.983964    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 13:15:38.985770 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.985711    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wqcfl\""
Sep 08 13:15:38.985886 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.985875    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 08 13:15:38.986027 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.986002    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-br278"
Sep 08 13:15:38.986085 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.986074    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-var-lib-openvswitch\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:38.986126 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:38.986086    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-br278" podUID="6fa64a88-f23c-42a6-8a1f-748e0278757a"
Sep 08 13:15:38.986126 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.986102    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-host-cni-netd\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:38.986315 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.986128    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-cnibin\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:38.986315 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.986208    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2913fac9-49a5-45a3-9181-b2d644239547-cni-binary-copy\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:38.986315 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.986260    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-host-var-lib-cni-bin\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:38.986315 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.986300    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvhkw\" (UniqueName: \"kubernetes.io/projected/74540ccb-b9a8-424b-bae1-5d7cd5ce07fa-kube-api-access-kvhkw\") pod \"multus-additional-cni-plugins-jtgss\" (UID: \"74540ccb-b9a8-424b-bae1-5d7cd5ce07fa\") " pod="openshift-multus/multus-additional-cni-plugins-jtgss"
Sep 08 13:15:38.986528 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.986333    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90dadd3f-c403-4d64-af3a-45b4c68163e5-metrics-certs\") pod \"network-metrics-daemon-q4m28\" (UID: \"90dadd3f-c403-4d64-af3a-45b4c68163e5\") " pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:15:38.986528 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.986368    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw5f5\" (UniqueName: \"kubernetes.io/projected/eb60fb08-544c-4bd8-93fe-103f5f2bde69-kube-api-access-tw5f5\") pod \"node-ca-7t44n\" (UID: \"eb60fb08-544c-4bd8-93fe-103f5f2bde69\") " pod="openshift-image-registry/node-ca-7t44n"
Sep 08 13:15:38.986528 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.986393    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-host-kubelet\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:38.986528 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.986411    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-log-socket\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:38.986528 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.986434    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-host-cni-bin\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:38.986528 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.986467    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/97e67a4f-0d9f-4646-a1e5-2b12014a6242-env-overrides\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:38.986528 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.986498    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-system-cni-dir\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:38.986528 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.986516    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 08 13:15:38.986957 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.986522    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-multus-conf-dir\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:38.986957 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.986568    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2913fac9-49a5-45a3-9181-b2d644239547-multus-daemon-config\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:38.986957 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.986596    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-systemd-units\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:38.986957 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.986636    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-host-run-netns\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:38.986957 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.986661    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/97e67a4f-0d9f-4646-a1e5-2b12014a6242-ovnkube-script-lib\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:38.986957 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.986684    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-multus-cni-dir\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:38.986957 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.986700    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-multus-socket-dir-parent\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:38.986957 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.986715    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-host-var-lib-cni-multus\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:38.986957 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.986732    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-host-run-multus-certs\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:38.986957 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.986762    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-run-ovn\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:38.986957 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.986796    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eb60fb08-544c-4bd8-93fe-103f5f2bde69-serviceca\") pod \"node-ca-7t44n\" (UID: \"eb60fb08-544c-4bd8-93fe-103f5f2bde69\") " pod="openshift-image-registry/node-ca-7t44n"
Sep 08 13:15:38.986957 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.986820    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-host-run-k8s-cni-cncf-io\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:38.986957 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.986845    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxkj4\" (UniqueName: \"kubernetes.io/projected/2913fac9-49a5-45a3-9181-b2d644239547-kube-api-access-wxkj4\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:38.986957 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.986881    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74540ccb-b9a8-424b-bae1-5d7cd5ce07fa-cnibin\") pod \"multus-additional-cni-plugins-jtgss\" (UID: \"74540ccb-b9a8-424b-bae1-5d7cd5ce07fa\") " pod="openshift-multus/multus-additional-cni-plugins-jtgss"
Sep 08 13:15:38.986957 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.986924    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74540ccb-b9a8-424b-bae1-5d7cd5ce07fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-jtgss\" (UID: \"74540ccb-b9a8-424b-bae1-5d7cd5ce07fa\") " pod="openshift-multus/multus-additional-cni-plugins-jtgss"
Sep 08 13:15:38.986957 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.986958    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74540ccb-b9a8-424b-bae1-5d7cd5ce07fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jtgss\" (UID: \"74540ccb-b9a8-424b-bae1-5d7cd5ce07fa\") " pod="openshift-multus/multus-additional-cni-plugins-jtgss"
Sep 08 13:15:38.987761 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.986984    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/74540ccb-b9a8-424b-bae1-5d7cd5ce07fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jtgss\" (UID: \"74540ccb-b9a8-424b-bae1-5d7cd5ce07fa\") " pod="openshift-multus/multus-additional-cni-plugins-jtgss"
Sep 08 13:15:38.987761 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.987011    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klvbz\" (UniqueName: \"kubernetes.io/projected/97e67a4f-0d9f-4646-a1e5-2b12014a6242-kube-api-access-klvbz\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:38.987761 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.987038    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb60fb08-544c-4bd8-93fe-103f5f2bde69-host\") pod \"node-ca-7t44n\" (UID: \"eb60fb08-544c-4bd8-93fe-103f5f2bde69\") " pod="openshift-image-registry/node-ca-7t44n"
Sep 08 13:15:38.987761 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.987063    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-run-openvswitch\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:38.987761 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.987087    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-node-log\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:38.987761 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.987139    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-host-slash\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:38.987761 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.987164    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-run-systemd\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:38.987761 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.987191    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-etc-openvswitch\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:38.987761 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.987237    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:38.987761 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.987271    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/97e67a4f-0d9f-4646-a1e5-2b12014a6242-ovnkube-config\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:38.987761 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.987299    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/97e67a4f-0d9f-4646-a1e5-2b12014a6242-ovn-node-metrics-cert\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:38.987761 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.987329    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-host-run-netns\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:38.987761 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.987353    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-host-var-lib-kubelet\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:38.987761 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.987377    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-hostroot\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:38.987761 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.987422    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-etc-kubernetes\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:38.987761 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.987461    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74540ccb-b9a8-424b-bae1-5d7cd5ce07fa-system-cni-dir\") pod \"multus-additional-cni-plugins-jtgss\" (UID: \"74540ccb-b9a8-424b-bae1-5d7cd5ce07fa\") " pod="openshift-multus/multus-additional-cni-plugins-jtgss"
Sep 08 13:15:38.988460 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.987487    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-host-run-ovn-kubernetes\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:38.988460 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.987515    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/74540ccb-b9a8-424b-bae1-5d7cd5ce07fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jtgss\" (UID: \"74540ccb-b9a8-424b-bae1-5d7cd5ce07fa\") " pod="openshift-multus/multus-additional-cni-plugins-jtgss"
Sep 08 13:15:38.988460 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.987554    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbflq\" (UniqueName: \"kubernetes.io/projected/90dadd3f-c403-4d64-af3a-45b4c68163e5-kube-api-access-dbflq\") pod \"network-metrics-daemon-q4m28\" (UID: \"90dadd3f-c403-4d64-af3a-45b4c68163e5\") " pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:15:38.988661 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.988645    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vwvwg"
Sep 08 13:15:38.990809 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.990787    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 08 13:15:38.990910 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.990845    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 08 13:15:38.990968 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.990931    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rlhvg\""
Sep 08 13:15:38.991423 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.991406    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5zws"
Sep 08 13:15:38.994156 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.993797    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 08 13:15:38.994156 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.993808    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 08 13:15:38.994156 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.993899    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:38.994156 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.993924    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 08 13:15:38.994500 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.994481    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h2ngk\""
Sep 08 13:15:38.997137 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.997117    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:15:38.997269 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:38.997184    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5jt7" podUID="24a4cf01-77e6-44a0-8074-0929ee8dfee7"
Sep 08 13:15:38.998582 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.998557    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5fg24\""
Sep 08 13:15:38.998702 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.998629    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:15:38.998839 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:38.998820    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 08 13:15:39.008656 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.008634    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-psz62"
Sep 08 13:15:39.017425 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.017402    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-psz62"
Sep 08 13:15:39.054244 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.054188    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-147.ec2.internal" event={"ID":"485a6b3955bdaff3eb439536885dfd93","Type":"ContainerStarted","Data":"1373d61b6922008b9102ef8450aaa687254a496643400c37118b4e620a4a592b"}
Sep 08 13:15:39.074000 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.073971    2558 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 08 13:15:39.088031 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.087992    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/eb97e09e-f5cf-48dd-a75c-f40e60d9756d-device-dir\") pod \"aws-ebs-csi-driver-node-m5zws\" (UID: \"eb97e09e-f5cf-48dd-a75c-f40e60d9756d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5zws"
Sep 08 13:15:39.088198 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088037    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d476dc00-0e12-46b0-900a-a95c96fd7ca3-etc-kubernetes\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.088198 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088074    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-run-ovn\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.088198 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088102    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eb60fb08-544c-4bd8-93fe-103f5f2bde69-serviceca\") pod \"node-ca-7t44n\" (UID: \"eb60fb08-544c-4bd8-93fe-103f5f2bde69\") " pod="openshift-image-registry/node-ca-7t44n"
Sep 08 13:15:39.088198 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088126    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxkj4\" (UniqueName: \"kubernetes.io/projected/2913fac9-49a5-45a3-9181-b2d644239547-kube-api-access-wxkj4\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:39.088198 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088146    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/eb97e09e-f5cf-48dd-a75c-f40e60d9756d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m5zws\" (UID: \"eb97e09e-f5cf-48dd-a75c-f40e60d9756d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5zws"
Sep 08 13:15:39.088198 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088170    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/eb97e09e-f5cf-48dd-a75c-f40e60d9756d-socket-dir\") pod \"aws-ebs-csi-driver-node-m5zws\" (UID: \"eb97e09e-f5cf-48dd-a75c-f40e60d9756d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5zws"
Sep 08 13:15:39.088198 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088193    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d477g\" (UniqueName: \"kubernetes.io/projected/eb97e09e-f5cf-48dd-a75c-f40e60d9756d-kube-api-access-d477g\") pod \"aws-ebs-csi-driver-node-m5zws\" (UID: \"eb97e09e-f5cf-48dd-a75c-f40e60d9756d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5zws"
Sep 08 13:15:39.088601 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088223    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24a4cf01-77e6-44a0-8074-0929ee8dfee7-original-pull-secret\") pod \"global-pull-secret-syncer-g5jt7\" (UID: \"24a4cf01-77e6-44a0-8074-0929ee8dfee7\") " pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:15:39.088601 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088249    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klvbz\" (UniqueName: \"kubernetes.io/projected/97e67a4f-0d9f-4646-a1e5-2b12014a6242-kube-api-access-klvbz\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.088601 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088248    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-run-ovn\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.088601 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088273    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-run-openvswitch\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.088601 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088300    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-node-log\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.088601 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088338    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-run-systemd\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.088601 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088355    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-node-log\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.088601 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088372    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-etc-openvswitch\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.088601 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088416    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-run-systemd\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.088601 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088463    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-run-openvswitch\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.088601 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088499    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-host-run-netns\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:39.088601 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088525    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-host-var-lib-kubelet\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:39.088601 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088567    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.088601 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088593    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/97e67a4f-0d9f-4646-a1e5-2b12014a6242-ovnkube-config\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.088601 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088594    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eb60fb08-544c-4bd8-93fe-103f5f2bde69-serviceca\") pod \"node-ca-7t44n\" (UID: \"eb60fb08-544c-4bd8-93fe-103f5f2bde69\") " pod="openshift-image-registry/node-ca-7t44n"
Sep 08 13:15:39.088601 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088606    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-host-var-lib-kubelet\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:39.089393 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088613    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-hostroot\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:39.089393 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088666    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74540ccb-b9a8-424b-bae1-5d7cd5ce07fa-system-cni-dir\") pod \"multus-additional-cni-plugins-jtgss\" (UID: \"74540ccb-b9a8-424b-bae1-5d7cd5ce07fa\") " pod="openshift-multus/multus-additional-cni-plugins-jtgss"
Sep 08 13:15:39.089393 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088691    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/74540ccb-b9a8-424b-bae1-5d7cd5ce07fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jtgss\" (UID: \"74540ccb-b9a8-424b-bae1-5d7cd5ce07fa\") " pod="openshift-multus/multus-additional-cni-plugins-jtgss"
Sep 08 13:15:39.089393 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088699    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-etc-openvswitch\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.089393 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088728    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-host-run-netns\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:39.089393 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088781    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-hostroot\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:39.089393 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088792    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74540ccb-b9a8-424b-bae1-5d7cd5ce07fa-system-cni-dir\") pod \"multus-additional-cni-plugins-jtgss\" (UID: \"74540ccb-b9a8-424b-bae1-5d7cd5ce07fa\") " pod="openshift-multus/multus-additional-cni-plugins-jtgss"
Sep 08 13:15:39.089393 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088805    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.089393 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088834    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbflq\" (UniqueName: \"kubernetes.io/projected/90dadd3f-c403-4d64-af3a-45b4c68163e5-kube-api-access-dbflq\") pod \"network-metrics-daemon-q4m28\" (UID: \"90dadd3f-c403-4d64-af3a-45b4c68163e5\") " pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:15:39.089393 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088871    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90dadd3f-c403-4d64-af3a-45b4c68163e5-metrics-certs\") pod \"network-metrics-daemon-q4m28\" (UID: \"90dadd3f-c403-4d64-af3a-45b4c68163e5\") " pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:15:39.089393 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.088986    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d476dc00-0e12-46b0-900a-a95c96fd7ca3-etc-sysctl-d\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.089393 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:39.089030    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:15:39.089393 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:39.089137    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90dadd3f-c403-4d64-af3a-45b4c68163e5-metrics-certs podName:90dadd3f-c403-4d64-af3a-45b4c68163e5 nodeName:}" failed. No retries permitted until 2026-09-08 13:15:39.589111285 +0000 UTC m=+3.040872197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90dadd3f-c403-4d64-af3a-45b4c68163e5-metrics-certs") pod "network-metrics-daemon-q4m28" (UID: "90dadd3f-c403-4d64-af3a-45b4c68163e5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:15:39.089393 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089161    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d476dc00-0e12-46b0-900a-a95c96fd7ca3-etc-sysconfig\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.089393 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089200    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d476dc00-0e12-46b0-900a-a95c96fd7ca3-host\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.089393 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089224    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/97e67a4f-0d9f-4646-a1e5-2b12014a6242-ovnkube-config\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.089393 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089227    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/24a4cf01-77e6-44a0-8074-0929ee8dfee7-kubelet-config\") pod \"global-pull-secret-syncer-g5jt7\" (UID: \"24a4cf01-77e6-44a0-8074-0929ee8dfee7\") " pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:15:39.090182 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089285    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-var-lib-openvswitch\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.090182 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089307    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-host-cni-netd\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.090182 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089336    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-host-var-lib-cni-bin\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:39.090182 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089360    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-var-lib-openvswitch\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.090182 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089362    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3bb8c2d9-5301-4487-a874-f2825ab4f458-host-slash\") pod \"iptables-alerter-75fqq\" (UID: \"3bb8c2d9-5301-4487-a874-f2825ab4f458\") " pod="openshift-network-operator/iptables-alerter-75fqq"
Sep 08 13:15:39.090182 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089406    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/eb97e09e-f5cf-48dd-a75c-f40e60d9756d-sys-fs\") pod \"aws-ebs-csi-driver-node-m5zws\" (UID: \"eb97e09e-f5cf-48dd-a75c-f40e60d9756d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5zws"
Sep 08 13:15:39.090182 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089429    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d476dc00-0e12-46b0-900a-a95c96fd7ca3-etc-systemd\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.090182 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089445    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d476dc00-0e12-46b0-900a-a95c96fd7ca3-etc-tuned\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.090182 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089439    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-host-cni-netd\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.090182 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089467    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw5f5\" (UniqueName: \"kubernetes.io/projected/eb60fb08-544c-4bd8-93fe-103f5f2bde69-kube-api-access-tw5f5\") pod \"node-ca-7t44n\" (UID: \"eb60fb08-544c-4bd8-93fe-103f5f2bde69\") " pod="openshift-image-registry/node-ca-7t44n"
Sep 08 13:15:39.090182 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089493    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkd5f\" (UniqueName: \"kubernetes.io/projected/6fa64a88-f23c-42a6-8a1f-748e0278757a-kube-api-access-wkd5f\") pod \"network-check-target-br278\" (UID: \"6fa64a88-f23c-42a6-8a1f-748e0278757a\") " pod="openshift-network-diagnostics/network-check-target-br278"
Sep 08 13:15:39.090182 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089504    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-host-var-lib-cni-bin\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:39.090182 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089523    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d476dc00-0e12-46b0-900a-a95c96fd7ca3-lib-modules\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.090182 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089547    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-log-socket\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.090182 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089572    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-host-cni-bin\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.090182 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089634    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-host-cni-bin\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.090182 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089634    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-log-socket\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.091012 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089655    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/97e67a4f-0d9f-4646-a1e5-2b12014a6242-env-overrides\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.091012 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089684    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2913fac9-49a5-45a3-9181-b2d644239547-multus-daemon-config\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:39.091012 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089708    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ce2ec68-0166-484d-8e1a-28cc99334e5d-tmp-dir\") pod \"node-resolver-vwvwg\" (UID: \"3ce2ec68-0166-484d-8e1a-28cc99334e5d\") " pod="openshift-dns/node-resolver-vwvwg"
Sep 08 13:15:39.091012 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089738    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-host-run-netns\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.091012 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089768    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-multus-cni-dir\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:39.091012 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089809    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-multus-socket-dir-parent\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:39.091012 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089846    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3bb8c2d9-5301-4487-a874-f2825ab4f458-iptables-alerter-script\") pod \"iptables-alerter-75fqq\" (UID: \"3bb8c2d9-5301-4487-a874-f2825ab4f458\") " pod="openshift-network-operator/iptables-alerter-75fqq"
Sep 08 13:15:39.091012 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089876    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-host-run-k8s-cni-cncf-io\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:39.091012 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089900    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74540ccb-b9a8-424b-bae1-5d7cd5ce07fa-cnibin\") pod \"multus-additional-cni-plugins-jtgss\" (UID: \"74540ccb-b9a8-424b-bae1-5d7cd5ce07fa\") " pod="openshift-multus/multus-additional-cni-plugins-jtgss"
Sep 08 13:15:39.091012 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089916    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74540ccb-b9a8-424b-bae1-5d7cd5ce07fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-jtgss\" (UID: \"74540ccb-b9a8-424b-bae1-5d7cd5ce07fa\") " pod="openshift-multus/multus-additional-cni-plugins-jtgss"
Sep 08 13:15:39.091012 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089933    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74540ccb-b9a8-424b-bae1-5d7cd5ce07fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jtgss\" (UID: \"74540ccb-b9a8-424b-bae1-5d7cd5ce07fa\") " pod="openshift-multus/multus-additional-cni-plugins-jtgss"
Sep 08 13:15:39.091012 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089951    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/58dbf292-5174-4712-b380-d5763868b1c5-konnectivity-ca\") pod \"konnectivity-agent-k9f94\" (UID: \"58dbf292-5174-4712-b380-d5763868b1c5\") " pod="kube-system/konnectivity-agent-k9f94"
Sep 08 13:15:39.091012 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089968    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb60fb08-544c-4bd8-93fe-103f5f2bde69-host\") pod \"node-ca-7t44n\" (UID: \"eb60fb08-544c-4bd8-93fe-103f5f2bde69\") " pod="openshift-image-registry/node-ca-7t44n"
Sep 08 13:15:39.091012 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089984    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-host-slash\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.091012 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.089996    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-multus-cni-dir\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:39.091012 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090000    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/97e67a4f-0d9f-4646-a1e5-2b12014a6242-ovn-node-metrics-cert\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.091012 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090040    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d476dc00-0e12-46b0-900a-a95c96fd7ca3-tmp\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.091917 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090058    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/97e67a4f-0d9f-4646-a1e5-2b12014a6242-env-overrides\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.091917 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090076    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-etc-kubernetes\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:39.091917 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090116    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-host-run-netns\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.091917 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090126    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-etc-kubernetes\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:39.091917 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090148    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-host-run-ovn-kubernetes\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.091917 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090189    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-host-run-ovn-kubernetes\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.091917 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090205    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/74540ccb-b9a8-424b-bae1-5d7cd5ce07fa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jtgss\" (UID: \"74540ccb-b9a8-424b-bae1-5d7cd5ce07fa\") " pod="openshift-multus/multus-additional-cni-plugins-jtgss"
Sep 08 13:15:39.091917 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090281    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-host-run-k8s-cni-cncf-io\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:39.091917 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090287    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-host-slash\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.091917 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090276    2558 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 08 13:15:39.091917 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090306    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/58dbf292-5174-4712-b380-d5763868b1c5-agent-certs\") pod \"konnectivity-agent-k9f94\" (UID: \"58dbf292-5174-4712-b380-d5763868b1c5\") " pod="kube-system/konnectivity-agent-k9f94"
Sep 08 13:15:39.091917 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090327    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb60fb08-544c-4bd8-93fe-103f5f2bde69-host\") pod \"node-ca-7t44n\" (UID: \"eb60fb08-544c-4bd8-93fe-103f5f2bde69\") " pod="openshift-image-registry/node-ca-7t44n"
Sep 08 13:15:39.091917 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090353    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-multus-socket-dir-parent\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:39.091917 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090355    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/eb97e09e-f5cf-48dd-a75c-f40e60d9756d-registration-dir\") pod \"aws-ebs-csi-driver-node-m5zws\" (UID: \"eb97e09e-f5cf-48dd-a75c-f40e60d9756d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5zws"
Sep 08 13:15:39.091917 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090360    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74540ccb-b9a8-424b-bae1-5d7cd5ce07fa-cnibin\") pod \"multus-additional-cni-plugins-jtgss\" (UID: \"74540ccb-b9a8-424b-bae1-5d7cd5ce07fa\") " pod="openshift-multus/multus-additional-cni-plugins-jtgss"
Sep 08 13:15:39.091917 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090359    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74540ccb-b9a8-424b-bae1-5d7cd5ce07fa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jtgss\" (UID: \"74540ccb-b9a8-424b-bae1-5d7cd5ce07fa\") " pod="openshift-multus/multus-additional-cni-plugins-jtgss"
Sep 08 13:15:39.091917 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090394    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/eb97e09e-f5cf-48dd-a75c-f40e60d9756d-etc-selinux\") pod \"aws-ebs-csi-driver-node-m5zws\" (UID: \"eb97e09e-f5cf-48dd-a75c-f40e60d9756d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5zws"
Sep 08 13:15:39.092639 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090422    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d476dc00-0e12-46b0-900a-a95c96fd7ca3-var-lib-kubelet\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.092639 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090453    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-cnibin\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:39.092639 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090478    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2913fac9-49a5-45a3-9181-b2d644239547-cni-binary-copy\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:39.092639 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090507    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvhkw\" (UniqueName: \"kubernetes.io/projected/74540ccb-b9a8-424b-bae1-5d7cd5ce07fa-kube-api-access-kvhkw\") pod \"multus-additional-cni-plugins-jtgss\" (UID: \"74540ccb-b9a8-424b-bae1-5d7cd5ce07fa\") " pod="openshift-multus/multus-additional-cni-plugins-jtgss"
Sep 08 13:15:39.092639 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090533    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d476dc00-0e12-46b0-900a-a95c96fd7ca3-run\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.092639 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090537    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-cnibin\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:39.092639 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090556    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/24a4cf01-77e6-44a0-8074-0929ee8dfee7-dbus\") pod \"global-pull-secret-syncer-g5jt7\" (UID: \"24a4cf01-77e6-44a0-8074-0929ee8dfee7\") " pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:15:39.092639 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090583    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-host-kubelet\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.092639 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090595    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74540ccb-b9a8-424b-bae1-5d7cd5ce07fa-cni-binary-copy\") pod \"multus-additional-cni-plugins-jtgss\" (UID: \"74540ccb-b9a8-424b-bae1-5d7cd5ce07fa\") " pod="openshift-multus/multus-additional-cni-plugins-jtgss"
Sep 08 13:15:39.092639 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090616    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3ce2ec68-0166-484d-8e1a-28cc99334e5d-hosts-file\") pod \"node-resolver-vwvwg\" (UID: \"3ce2ec68-0166-484d-8e1a-28cc99334e5d\") " pod="openshift-dns/node-resolver-vwvwg"
Sep 08 13:15:39.092639 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090638    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gnhf\" (UniqueName: \"kubernetes.io/projected/3bb8c2d9-5301-4487-a874-f2825ab4f458-kube-api-access-7gnhf\") pod \"iptables-alerter-75fqq\" (UID: \"3bb8c2d9-5301-4487-a874-f2825ab4f458\") " pod="openshift-network-operator/iptables-alerter-75fqq"
Sep 08 13:15:39.092639 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090681    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d476dc00-0e12-46b0-900a-a95c96fd7ca3-etc-modprobe-d\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.092639 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090698    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-host-kubelet\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.092639 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090707    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mksrl\" (UniqueName: \"kubernetes.io/projected/d476dc00-0e12-46b0-900a-a95c96fd7ca3-kube-api-access-mksrl\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.092639 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090729    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-system-cni-dir\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:39.092639 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090764    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-multus-conf-dir\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:39.092639 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090803    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxqrm\" (UniqueName: \"kubernetes.io/projected/3ce2ec68-0166-484d-8e1a-28cc99334e5d-kube-api-access-lxqrm\") pod \"node-resolver-vwvwg\" (UID: \"3ce2ec68-0166-484d-8e1a-28cc99334e5d\") " pod="openshift-dns/node-resolver-vwvwg"
Sep 08 13:15:39.093449 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090813    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-multus-conf-dir\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:39.093449 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090834    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-system-cni-dir\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:39.093449 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090837    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d476dc00-0e12-46b0-900a-a95c96fd7ca3-etc-sysctl-conf\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.093449 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090867    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d476dc00-0e12-46b0-900a-a95c96fd7ca3-sys\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.093449 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090896    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-systemd-units\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.093449 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090922    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/97e67a4f-0d9f-4646-a1e5-2b12014a6242-ovnkube-script-lib\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.093449 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090938    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/97e67a4f-0d9f-4646-a1e5-2b12014a6242-systemd-units\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.093449 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090949    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-host-var-lib-cni-multus\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:39.093449 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090990    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2913fac9-49a5-45a3-9181-b2d644239547-cni-binary-copy\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:39.093449 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.091014    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-host-run-multus-certs\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:39.093449 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.091047    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-host-run-multus-certs\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:39.093449 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.090985    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2913fac9-49a5-45a3-9181-b2d644239547-host-var-lib-cni-multus\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:39.093449 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.091064    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/74540ccb-b9a8-424b-bae1-5d7cd5ce07fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jtgss\" (UID: \"74540ccb-b9a8-424b-bae1-5d7cd5ce07fa\") " pod="openshift-multus/multus-additional-cni-plugins-jtgss"
Sep 08 13:15:39.093449 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.091152    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2913fac9-49a5-45a3-9181-b2d644239547-multus-daemon-config\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:39.093449 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.091440    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/97e67a4f-0d9f-4646-a1e5-2b12014a6242-ovnkube-script-lib\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.093449 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.091539    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/74540ccb-b9a8-424b-bae1-5d7cd5ce07fa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jtgss\" (UID: \"74540ccb-b9a8-424b-bae1-5d7cd5ce07fa\") " pod="openshift-multus/multus-additional-cni-plugins-jtgss"
Sep 08 13:15:39.094764 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.094743    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/97e67a4f-0d9f-4646-a1e5-2b12014a6242-ovn-node-metrics-cert\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.098579 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.098421    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klvbz\" (UniqueName: \"kubernetes.io/projected/97e67a4f-0d9f-4646-a1e5-2b12014a6242-kube-api-access-klvbz\") pod \"ovnkube-node-tzwrm\" (UID: \"97e67a4f-0d9f-4646-a1e5-2b12014a6242\") " pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.099027 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.098956    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxkj4\" (UniqueName: \"kubernetes.io/projected/2913fac9-49a5-45a3-9181-b2d644239547-kube-api-access-wxkj4\") pod \"multus-zss2s\" (UID: \"2913fac9-49a5-45a3-9181-b2d644239547\") " pod="openshift-multus/multus-zss2s"
Sep 08 13:15:39.099027 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.098982    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbflq\" (UniqueName: \"kubernetes.io/projected/90dadd3f-c403-4d64-af3a-45b4c68163e5-kube-api-access-dbflq\") pod \"network-metrics-daemon-q4m28\" (UID: \"90dadd3f-c403-4d64-af3a-45b4c68163e5\") " pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:15:39.099890 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.099750    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw5f5\" (UniqueName: \"kubernetes.io/projected/eb60fb08-544c-4bd8-93fe-103f5f2bde69-kube-api-access-tw5f5\") pod \"node-ca-7t44n\" (UID: \"eb60fb08-544c-4bd8-93fe-103f5f2bde69\") " pod="openshift-image-registry/node-ca-7t44n"
Sep 08 13:15:39.099890 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.099821    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvhkw\" (UniqueName: \"kubernetes.io/projected/74540ccb-b9a8-424b-bae1-5d7cd5ce07fa-kube-api-access-kvhkw\") pod \"multus-additional-cni-plugins-jtgss\" (UID: \"74540ccb-b9a8-424b-bae1-5d7cd5ce07fa\") " pod="openshift-multus/multus-additional-cni-plugins-jtgss"
Sep 08 13:15:39.191854 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.191824    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d476dc00-0e12-46b0-900a-a95c96fd7ca3-run\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.191854 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.191854    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/24a4cf01-77e6-44a0-8074-0929ee8dfee7-dbus\") pod \"global-pull-secret-syncer-g5jt7\" (UID: \"24a4cf01-77e6-44a0-8074-0929ee8dfee7\") " pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:15:39.192113 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.191873    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3ce2ec68-0166-484d-8e1a-28cc99334e5d-hosts-file\") pod \"node-resolver-vwvwg\" (UID: \"3ce2ec68-0166-484d-8e1a-28cc99334e5d\") " pod="openshift-dns/node-resolver-vwvwg"
Sep 08 13:15:39.192113 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.191889    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gnhf\" (UniqueName: \"kubernetes.io/projected/3bb8c2d9-5301-4487-a874-f2825ab4f458-kube-api-access-7gnhf\") pod \"iptables-alerter-75fqq\" (UID: \"3bb8c2d9-5301-4487-a874-f2825ab4f458\") " pod="openshift-network-operator/iptables-alerter-75fqq"
Sep 08 13:15:39.192113 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.191916    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d476dc00-0e12-46b0-900a-a95c96fd7ca3-etc-modprobe-d\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.192113 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.191934    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mksrl\" (UniqueName: \"kubernetes.io/projected/d476dc00-0e12-46b0-900a-a95c96fd7ca3-kube-api-access-mksrl\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.192113 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.191955    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxqrm\" (UniqueName: \"kubernetes.io/projected/3ce2ec68-0166-484d-8e1a-28cc99334e5d-kube-api-access-lxqrm\") pod \"node-resolver-vwvwg\" (UID: \"3ce2ec68-0166-484d-8e1a-28cc99334e5d\") " pod="openshift-dns/node-resolver-vwvwg"
Sep 08 13:15:39.192113 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.191979    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d476dc00-0e12-46b0-900a-a95c96fd7ca3-etc-sysctl-conf\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.192113 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.191988    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d476dc00-0e12-46b0-900a-a95c96fd7ca3-run\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.192113 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192006    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d476dc00-0e12-46b0-900a-a95c96fd7ca3-sys\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.192113 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192030    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3ce2ec68-0166-484d-8e1a-28cc99334e5d-hosts-file\") pod \"node-resolver-vwvwg\" (UID: \"3ce2ec68-0166-484d-8e1a-28cc99334e5d\") " pod="openshift-dns/node-resolver-vwvwg"
Sep 08 13:15:39.192113 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.191988    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/24a4cf01-77e6-44a0-8074-0929ee8dfee7-dbus\") pod \"global-pull-secret-syncer-g5jt7\" (UID: \"24a4cf01-77e6-44a0-8074-0929ee8dfee7\") " pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:15:39.192113 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192070    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d476dc00-0e12-46b0-900a-a95c96fd7ca3-etc-modprobe-d\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.192113 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192076    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d476dc00-0e12-46b0-900a-a95c96fd7ca3-sys\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.192584 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192145    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d476dc00-0e12-46b0-900a-a95c96fd7ca3-etc-sysctl-conf\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.192584 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192162    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/eb97e09e-f5cf-48dd-a75c-f40e60d9756d-device-dir\") pod \"aws-ebs-csi-driver-node-m5zws\" (UID: \"eb97e09e-f5cf-48dd-a75c-f40e60d9756d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5zws"
Sep 08 13:15:39.192584 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192195    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d476dc00-0e12-46b0-900a-a95c96fd7ca3-etc-kubernetes\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.192584 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192226    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/eb97e09e-f5cf-48dd-a75c-f40e60d9756d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m5zws\" (UID: \"eb97e09e-f5cf-48dd-a75c-f40e60d9756d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5zws"
Sep 08 13:15:39.192584 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192253    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/eb97e09e-f5cf-48dd-a75c-f40e60d9756d-socket-dir\") pod \"aws-ebs-csi-driver-node-m5zws\" (UID: \"eb97e09e-f5cf-48dd-a75c-f40e60d9756d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5zws"
Sep 08 13:15:39.192584 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192256    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/eb97e09e-f5cf-48dd-a75c-f40e60d9756d-device-dir\") pod \"aws-ebs-csi-driver-node-m5zws\" (UID: \"eb97e09e-f5cf-48dd-a75c-f40e60d9756d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5zws"
Sep 08 13:15:39.192584 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192265    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d476dc00-0e12-46b0-900a-a95c96fd7ca3-etc-kubernetes\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.192584 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192279    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d477g\" (UniqueName: \"kubernetes.io/projected/eb97e09e-f5cf-48dd-a75c-f40e60d9756d-kube-api-access-d477g\") pod \"aws-ebs-csi-driver-node-m5zws\" (UID: \"eb97e09e-f5cf-48dd-a75c-f40e60d9756d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5zws"
Sep 08 13:15:39.192584 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192295    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/eb97e09e-f5cf-48dd-a75c-f40e60d9756d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m5zws\" (UID: \"eb97e09e-f5cf-48dd-a75c-f40e60d9756d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5zws"
Sep 08 13:15:39.192584 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192326    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24a4cf01-77e6-44a0-8074-0929ee8dfee7-original-pull-secret\") pod \"global-pull-secret-syncer-g5jt7\" (UID: \"24a4cf01-77e6-44a0-8074-0929ee8dfee7\") " pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:15:39.192584 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192383    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d476dc00-0e12-46b0-900a-a95c96fd7ca3-etc-sysctl-d\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.192584 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192383    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/eb97e09e-f5cf-48dd-a75c-f40e60d9756d-socket-dir\") pod \"aws-ebs-csi-driver-node-m5zws\" (UID: \"eb97e09e-f5cf-48dd-a75c-f40e60d9756d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5zws"
Sep 08 13:15:39.192584 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192409    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d476dc00-0e12-46b0-900a-a95c96fd7ca3-etc-sysconfig\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.192584 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192445    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d476dc00-0e12-46b0-900a-a95c96fd7ca3-host\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.192584 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192450    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d476dc00-0e12-46b0-900a-a95c96fd7ca3-etc-sysconfig\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.192584 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:39.192454    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:15:39.192584 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192469    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/24a4cf01-77e6-44a0-8074-0929ee8dfee7-kubelet-config\") pod \"global-pull-secret-syncer-g5jt7\" (UID: \"24a4cf01-77e6-44a0-8074-0929ee8dfee7\") " pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:15:39.192584 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192485    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d476dc00-0e12-46b0-900a-a95c96fd7ca3-etc-sysctl-d\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.193433 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:39.192501    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a4cf01-77e6-44a0-8074-0929ee8dfee7-original-pull-secret podName:24a4cf01-77e6-44a0-8074-0929ee8dfee7 nodeName:}" failed. No retries permitted until 2026-09-08 13:15:39.692487374 +0000 UTC m=+3.144248283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24a4cf01-77e6-44a0-8074-0929ee8dfee7-original-pull-secret") pod "global-pull-secret-syncer-g5jt7" (UID: "24a4cf01-77e6-44a0-8074-0929ee8dfee7") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:15:39.193433 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192485    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d476dc00-0e12-46b0-900a-a95c96fd7ca3-host\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.193433 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192518    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3bb8c2d9-5301-4487-a874-f2825ab4f458-host-slash\") pod \"iptables-alerter-75fqq\" (UID: \"3bb8c2d9-5301-4487-a874-f2825ab4f458\") " pod="openshift-network-operator/iptables-alerter-75fqq"
Sep 08 13:15:39.193433 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192544    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/eb97e09e-f5cf-48dd-a75c-f40e60d9756d-sys-fs\") pod \"aws-ebs-csi-driver-node-m5zws\" (UID: \"eb97e09e-f5cf-48dd-a75c-f40e60d9756d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5zws"
Sep 08 13:15:39.193433 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192550    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/24a4cf01-77e6-44a0-8074-0929ee8dfee7-kubelet-config\") pod \"global-pull-secret-syncer-g5jt7\" (UID: \"24a4cf01-77e6-44a0-8074-0929ee8dfee7\") " pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:15:39.193433 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192569    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d476dc00-0e12-46b0-900a-a95c96fd7ca3-etc-systemd\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.193433 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192592    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d476dc00-0e12-46b0-900a-a95c96fd7ca3-etc-tuned\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.193433 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192609    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3bb8c2d9-5301-4487-a874-f2825ab4f458-host-slash\") pod \"iptables-alerter-75fqq\" (UID: \"3bb8c2d9-5301-4487-a874-f2825ab4f458\") " pod="openshift-network-operator/iptables-alerter-75fqq"
Sep 08 13:15:39.193433 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192624    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/eb97e09e-f5cf-48dd-a75c-f40e60d9756d-sys-fs\") pod \"aws-ebs-csi-driver-node-m5zws\" (UID: \"eb97e09e-f5cf-48dd-a75c-f40e60d9756d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5zws"
Sep 08 13:15:39.193433 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192638    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkd5f\" (UniqueName: \"kubernetes.io/projected/6fa64a88-f23c-42a6-8a1f-748e0278757a-kube-api-access-wkd5f\") pod \"network-check-target-br278\" (UID: \"6fa64a88-f23c-42a6-8a1f-748e0278757a\") " pod="openshift-network-diagnostics/network-check-target-br278"
Sep 08 13:15:39.193433 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192659    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d476dc00-0e12-46b0-900a-a95c96fd7ca3-etc-systemd\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.193433 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192670    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d476dc00-0e12-46b0-900a-a95c96fd7ca3-lib-modules\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.193433 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192701    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ce2ec68-0166-484d-8e1a-28cc99334e5d-tmp-dir\") pod \"node-resolver-vwvwg\" (UID: \"3ce2ec68-0166-484d-8e1a-28cc99334e5d\") " pod="openshift-dns/node-resolver-vwvwg"
Sep 08 13:15:39.193433 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192730    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3bb8c2d9-5301-4487-a874-f2825ab4f458-iptables-alerter-script\") pod \"iptables-alerter-75fqq\" (UID: \"3bb8c2d9-5301-4487-a874-f2825ab4f458\") " pod="openshift-network-operator/iptables-alerter-75fqq"
Sep 08 13:15:39.193433 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192765    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/58dbf292-5174-4712-b380-d5763868b1c5-konnectivity-ca\") pod \"konnectivity-agent-k9f94\" (UID: \"58dbf292-5174-4712-b380-d5763868b1c5\") " pod="kube-system/konnectivity-agent-k9f94"
Sep 08 13:15:39.193433 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192789    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d476dc00-0e12-46b0-900a-a95c96fd7ca3-lib-modules\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.193433 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192792    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d476dc00-0e12-46b0-900a-a95c96fd7ca3-tmp\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.194005 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192914    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/58dbf292-5174-4712-b380-d5763868b1c5-agent-certs\") pod \"konnectivity-agent-k9f94\" (UID: \"58dbf292-5174-4712-b380-d5763868b1c5\") " pod="kube-system/konnectivity-agent-k9f94"
Sep 08 13:15:39.194005 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.192953    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/eb97e09e-f5cf-48dd-a75c-f40e60d9756d-registration-dir\") pod \"aws-ebs-csi-driver-node-m5zws\" (UID: \"eb97e09e-f5cf-48dd-a75c-f40e60d9756d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5zws"
Sep 08 13:15:39.194005 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.193018    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/eb97e09e-f5cf-48dd-a75c-f40e60d9756d-etc-selinux\") pod \"aws-ebs-csi-driver-node-m5zws\" (UID: \"eb97e09e-f5cf-48dd-a75c-f40e60d9756d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5zws"
Sep 08 13:15:39.194005 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.193046    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d476dc00-0e12-46b0-900a-a95c96fd7ca3-var-lib-kubelet\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.194005 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.193047    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ce2ec68-0166-484d-8e1a-28cc99334e5d-tmp-dir\") pod \"node-resolver-vwvwg\" (UID: \"3ce2ec68-0166-484d-8e1a-28cc99334e5d\") " pod="openshift-dns/node-resolver-vwvwg"
Sep 08 13:15:39.194005 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.193138    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/eb97e09e-f5cf-48dd-a75c-f40e60d9756d-etc-selinux\") pod \"aws-ebs-csi-driver-node-m5zws\" (UID: \"eb97e09e-f5cf-48dd-a75c-f40e60d9756d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5zws"
Sep 08 13:15:39.194005 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.193152    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d476dc00-0e12-46b0-900a-a95c96fd7ca3-var-lib-kubelet\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.194005 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.193194    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/eb97e09e-f5cf-48dd-a75c-f40e60d9756d-registration-dir\") pod \"aws-ebs-csi-driver-node-m5zws\" (UID: \"eb97e09e-f5cf-48dd-a75c-f40e60d9756d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5zws"
Sep 08 13:15:39.194005 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.193427    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3bb8c2d9-5301-4487-a874-f2825ab4f458-iptables-alerter-script\") pod \"iptables-alerter-75fqq\" (UID: \"3bb8c2d9-5301-4487-a874-f2825ab4f458\") " pod="openshift-network-operator/iptables-alerter-75fqq"
Sep 08 13:15:39.194005 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.193425    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/58dbf292-5174-4712-b380-d5763868b1c5-konnectivity-ca\") pod \"konnectivity-agent-k9f94\" (UID: \"58dbf292-5174-4712-b380-d5763868b1c5\") " pod="kube-system/konnectivity-agent-k9f94"
Sep 08 13:15:39.195072 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.195052    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d476dc00-0e12-46b0-900a-a95c96fd7ca3-etc-tuned\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.195175 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.195093    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d476dc00-0e12-46b0-900a-a95c96fd7ca3-tmp\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.195283 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.195265    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/58dbf292-5174-4712-b380-d5763868b1c5-agent-certs\") pod \"konnectivity-agent-k9f94\" (UID: \"58dbf292-5174-4712-b380-d5763868b1c5\") " pod="kube-system/konnectivity-agent-k9f94"
Sep 08 13:15:39.207113 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:39.207053    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:15:39.207113 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:39.207077    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:15:39.207113 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:39.207087    2558 projected.go:196] Error preparing data for projected volume kube-api-access-wkd5f for pod openshift-network-diagnostics/network-check-target-br278: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:15:39.207351 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:39.207169    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fa64a88-f23c-42a6-8a1f-748e0278757a-kube-api-access-wkd5f podName:6fa64a88-f23c-42a6-8a1f-748e0278757a nodeName:}" failed. No retries permitted until 2026-09-08 13:15:39.707155002 +0000 UTC m=+3.158915931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wkd5f" (UniqueName: "kubernetes.io/projected/6fa64a88-f23c-42a6-8a1f-748e0278757a-kube-api-access-wkd5f") pod "network-check-target-br278" (UID: "6fa64a88-f23c-42a6-8a1f-748e0278757a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:15:39.207556 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.207537    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxqrm\" (UniqueName: \"kubernetes.io/projected/3ce2ec68-0166-484d-8e1a-28cc99334e5d-kube-api-access-lxqrm\") pod \"node-resolver-vwvwg\" (UID: \"3ce2ec68-0166-484d-8e1a-28cc99334e5d\") " pod="openshift-dns/node-resolver-vwvwg"
Sep 08 13:15:39.208061 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.208037    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gnhf\" (UniqueName: \"kubernetes.io/projected/3bb8c2d9-5301-4487-a874-f2825ab4f458-kube-api-access-7gnhf\") pod \"iptables-alerter-75fqq\" (UID: \"3bb8c2d9-5301-4487-a874-f2825ab4f458\") " pod="openshift-network-operator/iptables-alerter-75fqq"
Sep 08 13:15:39.208173 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.208151    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mksrl\" (UniqueName: \"kubernetes.io/projected/d476dc00-0e12-46b0-900a-a95c96fd7ca3-kube-api-access-mksrl\") pod \"tuned-k7cdm\" (UID: \"d476dc00-0e12-46b0-900a-a95c96fd7ca3\") " pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.209383 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.209365    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d477g\" (UniqueName: \"kubernetes.io/projected/eb97e09e-f5cf-48dd-a75c-f40e60d9756d-kube-api-access-d477g\") pod \"aws-ebs-csi-driver-node-m5zws\" (UID: \"eb97e09e-f5cf-48dd-a75c-f40e60d9756d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5zws"
Sep 08 13:15:39.280534 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.280493    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:15:39.290463 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.290438    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7t44n"
Sep 08 13:15:39.291748 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:39.291716    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e67a4f_0d9f_4646_a1e5_2b12014a6242.slice/crio-10679e747cf5ef83024d7c96ae507efe933bfeffe729801962d4d001157d2055 WatchSource:0}: Error finding container 10679e747cf5ef83024d7c96ae507efe933bfeffe729801962d4d001157d2055: Status 404 returned error can't find the container with id 10679e747cf5ef83024d7c96ae507efe933bfeffe729801962d4d001157d2055
Sep 08 13:15:39.298384 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.298350    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zss2s"
Sep 08 13:15:39.299401 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:39.299377    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb60fb08_544c_4bd8_93fe_103f5f2bde69.slice/crio-c173a9c739c8cf2420767e557cb0cead43669f3a28b376f806dcbb1d1ed7a184 WatchSource:0}: Error finding container c173a9c739c8cf2420767e557cb0cead43669f3a28b376f806dcbb1d1ed7a184: Status 404 returned error can't find the container with id c173a9c739c8cf2420767e557cb0cead43669f3a28b376f806dcbb1d1ed7a184
Sep 08 13:15:39.304164 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.304135    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jtgss"
Sep 08 13:15:39.312832 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.312812    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-75fqq"
Sep 08 13:15:39.319963 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.319939    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k9f94"
Sep 08 13:15:39.326717 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.326697    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vwvwg"
Sep 08 13:15:39.334492 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.334475    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5zws"
Sep 08 13:15:39.340220 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.340196    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k7cdm"
Sep 08 13:15:39.596417 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.596325    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90dadd3f-c403-4d64-af3a-45b4c68163e5-metrics-certs\") pod \"network-metrics-daemon-q4m28\" (UID: \"90dadd3f-c403-4d64-af3a-45b4c68163e5\") " pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:15:39.596577 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:39.596463    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:15:39.596577 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:39.596541    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90dadd3f-c403-4d64-af3a-45b4c68163e5-metrics-certs podName:90dadd3f-c403-4d64-af3a-45b4c68163e5 nodeName:}" failed. No retries permitted until 2026-09-08 13:15:40.596520474 +0000 UTC m=+4.048281392 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90dadd3f-c403-4d64-af3a-45b4c68163e5-metrics-certs") pod "network-metrics-daemon-q4m28" (UID: "90dadd3f-c403-4d64-af3a-45b4c68163e5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:15:39.596714 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.596683    2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:15:39.650108 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:39.650077    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2913fac9_49a5_45a3_9181_b2d644239547.slice/crio-1877c1e19bede5c81dc4e5e31a73142162589fa0c10df4735b2e9487fb78bada WatchSource:0}: Error finding container 1877c1e19bede5c81dc4e5e31a73142162589fa0c10df4735b2e9487fb78bada: Status 404 returned error can't find the container with id 1877c1e19bede5c81dc4e5e31a73142162589fa0c10df4735b2e9487fb78bada
Sep 08 13:15:39.650670 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:39.650635    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce2ec68_0166_484d_8e1a_28cc99334e5d.slice/crio-24941aabdf1e4b3579205f0951472bc9b78e96e5f18225d74a7e1b5d133c15cb WatchSource:0}: Error finding container 24941aabdf1e4b3579205f0951472bc9b78e96e5f18225d74a7e1b5d133c15cb: Status 404 returned error can't find the container with id 24941aabdf1e4b3579205f0951472bc9b78e96e5f18225d74a7e1b5d133c15cb
Sep 08 13:15:39.652233 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:39.652140    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb8c2d9_5301_4487_a874_f2825ab4f458.slice/crio-b89f48fbf01ece59c5b9fcb1434ac62c7c471884a9800d6818f60973f40a495b WatchSource:0}: Error finding container b89f48fbf01ece59c5b9fcb1434ac62c7c471884a9800d6818f60973f40a495b: Status 404 returned error can't find the container with id b89f48fbf01ece59c5b9fcb1434ac62c7c471884a9800d6818f60973f40a495b
Sep 08 13:15:39.656337 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:39.656230    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58dbf292_5174_4712_b380_d5763868b1c5.slice/crio-bc5bf612f68b513b706151ab4ee76b05d19c748bc2da09beceb126e559e7610f WatchSource:0}: Error finding container bc5bf612f68b513b706151ab4ee76b05d19c748bc2da09beceb126e559e7610f: Status 404 returned error can't find the container with id bc5bf612f68b513b706151ab4ee76b05d19c748bc2da09beceb126e559e7610f
Sep 08 13:15:39.656809 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:39.656780    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74540ccb_b9a8_424b_bae1_5d7cd5ce07fa.slice/crio-ce0179a4715351a903b7765761685dba8d9846557b88c660b173f7059322d902 WatchSource:0}: Error finding container ce0179a4715351a903b7765761685dba8d9846557b88c660b173f7059322d902: Status 404 returned error can't find the container with id ce0179a4715351a903b7765761685dba8d9846557b88c660b173f7059322d902
Sep 08 13:15:39.657276 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:39.657244    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb97e09e_f5cf_48dd_a75c_f40e60d9756d.slice/crio-06e7571fb34c7acdfc9c66198d425cd9e6eab21b80170b894475068b4892ae0d WatchSource:0}: Error finding container 06e7571fb34c7acdfc9c66198d425cd9e6eab21b80170b894475068b4892ae0d: Status 404 returned error can't find the container with id 06e7571fb34c7acdfc9c66198d425cd9e6eab21b80170b894475068b4892ae0d
Sep 08 13:15:39.658546 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:39.658520    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd476dc00_0e12_46b0_900a_a95c96fd7ca3.slice/crio-2842cb4870e24336057bd3f9687dd7936b8b5953bd21c4731a53f5354fbd7c7c WatchSource:0}: Error finding container 2842cb4870e24336057bd3f9687dd7936b8b5953bd21c4731a53f5354fbd7c7c: Status 404 returned error can't find the container with id 2842cb4870e24336057bd3f9687dd7936b8b5953bd21c4731a53f5354fbd7c7c
Sep 08 13:15:39.697812 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.697782    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24a4cf01-77e6-44a0-8074-0929ee8dfee7-original-pull-secret\") pod \"global-pull-secret-syncer-g5jt7\" (UID: \"24a4cf01-77e6-44a0-8074-0929ee8dfee7\") " pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:15:39.697931 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:39.697914    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:15:39.697986 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:39.697975    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a4cf01-77e6-44a0-8074-0929ee8dfee7-original-pull-secret podName:24a4cf01-77e6-44a0-8074-0929ee8dfee7 nodeName:}" failed. No retries permitted until 2026-09-08 13:15:40.697960292 +0000 UTC m=+4.149721203 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24a4cf01-77e6-44a0-8074-0929ee8dfee7-original-pull-secret") pod "global-pull-secret-syncer-g5jt7" (UID: "24a4cf01-77e6-44a0-8074-0929ee8dfee7") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:15:39.799139 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:39.799095    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkd5f\" (UniqueName: \"kubernetes.io/projected/6fa64a88-f23c-42a6-8a1f-748e0278757a-kube-api-access-wkd5f\") pod \"network-check-target-br278\" (UID: \"6fa64a88-f23c-42a6-8a1f-748e0278757a\") " pod="openshift-network-diagnostics/network-check-target-br278"
Sep 08 13:15:39.799322 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:39.799275    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:15:39.799322 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:39.799309    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:15:39.799322 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:39.799323    2558 projected.go:196] Error preparing data for projected volume kube-api-access-wkd5f for pod openshift-network-diagnostics/network-check-target-br278: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:15:39.799482 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:39.799396    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fa64a88-f23c-42a6-8a1f-748e0278757a-kube-api-access-wkd5f podName:6fa64a88-f23c-42a6-8a1f-748e0278757a nodeName:}" failed. No retries permitted until 2026-09-08 13:15:40.799374837 +0000 UTC m=+4.251135764 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wkd5f" (UniqueName: "kubernetes.io/projected/6fa64a88-f23c-42a6-8a1f-748e0278757a-kube-api-access-wkd5f") pod "network-check-target-br278" (UID: "6fa64a88-f23c-42a6-8a1f-748e0278757a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:15:40.020856 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:40.020798    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 13:10:39 +0000 UTC" deadline="2028-04-30 15:05:25.1506895 +0000 UTC"
Sep 08 13:15:40.020856 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:40.020855    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14401h49m45.129838471s"
Sep 08 13:15:40.058157 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:40.058117    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vwvwg" event={"ID":"3ce2ec68-0166-484d-8e1a-28cc99334e5d","Type":"ContainerStarted","Data":"24941aabdf1e4b3579205f0951472bc9b78e96e5f18225d74a7e1b5d133c15cb"}
Sep 08 13:15:40.059707 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:40.059681    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zss2s" event={"ID":"2913fac9-49a5-45a3-9181-b2d644239547","Type":"ContainerStarted","Data":"1877c1e19bede5c81dc4e5e31a73142162589fa0c10df4735b2e9487fb78bada"}
Sep 08 13:15:40.060776 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:40.060751    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7t44n" event={"ID":"eb60fb08-544c-4bd8-93fe-103f5f2bde69","Type":"ContainerStarted","Data":"c173a9c739c8cf2420767e557cb0cead43669f3a28b376f806dcbb1d1ed7a184"}
Sep 08 13:15:40.061969 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:40.061949    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm" event={"ID":"97e67a4f-0d9f-4646-a1e5-2b12014a6242","Type":"ContainerStarted","Data":"10679e747cf5ef83024d7c96ae507efe933bfeffe729801962d4d001157d2055"}
Sep 08 13:15:40.063927 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:40.063906    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-147.ec2.internal" event={"ID":"485a6b3955bdaff3eb439536885dfd93","Type":"ContainerStarted","Data":"b1773f2aa33fe3104fbdf2a60839ee127e69d41b5a4d8a4a6b95bf91125013db"}
Sep 08 13:15:40.065031 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:40.065012    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k7cdm" event={"ID":"d476dc00-0e12-46b0-900a-a95c96fd7ca3","Type":"ContainerStarted","Data":"2842cb4870e24336057bd3f9687dd7936b8b5953bd21c4731a53f5354fbd7c7c"}
Sep 08 13:15:40.066030 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:40.066011    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5zws" event={"ID":"eb97e09e-f5cf-48dd-a75c-f40e60d9756d","Type":"ContainerStarted","Data":"06e7571fb34c7acdfc9c66198d425cd9e6eab21b80170b894475068b4892ae0d"}
Sep 08 13:15:40.067141 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:40.067118    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtgss" event={"ID":"74540ccb-b9a8-424b-bae1-5d7cd5ce07fa","Type":"ContainerStarted","Data":"ce0179a4715351a903b7765761685dba8d9846557b88c660b173f7059322d902"}
Sep 08 13:15:40.068296 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:40.068275    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k9f94" event={"ID":"58dbf292-5174-4712-b380-d5763868b1c5","Type":"ContainerStarted","Data":"bc5bf612f68b513b706151ab4ee76b05d19c748bc2da09beceb126e559e7610f"}
Sep 08 13:15:40.069439 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:40.069422    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-75fqq" event={"ID":"3bb8c2d9-5301-4487-a874-f2825ab4f458","Type":"ContainerStarted","Data":"b89f48fbf01ece59c5b9fcb1434ac62c7c471884a9800d6818f60973f40a495b"}
Sep 08 13:15:40.606105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:40.606059    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90dadd3f-c403-4d64-af3a-45b4c68163e5-metrics-certs\") pod \"network-metrics-daemon-q4m28\" (UID: \"90dadd3f-c403-4d64-af3a-45b4c68163e5\") " pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:15:40.606317 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:40.606244    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:15:40.606385 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:40.606319    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90dadd3f-c403-4d64-af3a-45b4c68163e5-metrics-certs podName:90dadd3f-c403-4d64-af3a-45b4c68163e5 nodeName:}" failed. No retries permitted until 2026-09-08 13:15:42.606298351 +0000 UTC m=+6.058059270 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90dadd3f-c403-4d64-af3a-45b4c68163e5-metrics-certs") pod "network-metrics-daemon-q4m28" (UID: "90dadd3f-c403-4d64-af3a-45b4c68163e5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:15:40.708065 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:40.707277    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24a4cf01-77e6-44a0-8074-0929ee8dfee7-original-pull-secret\") pod \"global-pull-secret-syncer-g5jt7\" (UID: \"24a4cf01-77e6-44a0-8074-0929ee8dfee7\") " pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:15:40.708065 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:40.707528    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:15:40.708065 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:40.707609    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a4cf01-77e6-44a0-8074-0929ee8dfee7-original-pull-secret podName:24a4cf01-77e6-44a0-8074-0929ee8dfee7 nodeName:}" failed. No retries permitted until 2026-09-08 13:15:42.707586582 +0000 UTC m=+6.159347495 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24a4cf01-77e6-44a0-8074-0929ee8dfee7-original-pull-secret") pod "global-pull-secret-syncer-g5jt7" (UID: "24a4cf01-77e6-44a0-8074-0929ee8dfee7") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:15:40.808868 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:40.808824    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkd5f\" (UniqueName: \"kubernetes.io/projected/6fa64a88-f23c-42a6-8a1f-748e0278757a-kube-api-access-wkd5f\") pod \"network-check-target-br278\" (UID: \"6fa64a88-f23c-42a6-8a1f-748e0278757a\") " pod="openshift-network-diagnostics/network-check-target-br278"
Sep 08 13:15:40.809123 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:40.808996    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:15:40.809123 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:40.809018    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:15:40.809123 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:40.809030    2558 projected.go:196] Error preparing data for projected volume kube-api-access-wkd5f for pod openshift-network-diagnostics/network-check-target-br278: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:15:40.809123 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:40.809088    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fa64a88-f23c-42a6-8a1f-748e0278757a-kube-api-access-wkd5f podName:6fa64a88-f23c-42a6-8a1f-748e0278757a nodeName:}" failed. No retries permitted until 2026-09-08 13:15:42.809070118 +0000 UTC m=+6.260831030 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wkd5f" (UniqueName: "kubernetes.io/projected/6fa64a88-f23c-42a6-8a1f-748e0278757a-kube-api-access-wkd5f") pod "network-check-target-br278" (UID: "6fa64a88-f23c-42a6-8a1f-748e0278757a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:15:41.021528 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:41.021480    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 13:10:39 +0000 UTC" deadline="2028-04-19 23:30:11.806196471 +0000 UTC"
Sep 08 13:15:41.021528 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:41.021526    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14146h14m30.784675336s"
Sep 08 13:15:41.048131 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:41.047403    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:15:41.048131 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:41.047545    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4m28" podUID="90dadd3f-c403-4d64-af3a-45b4c68163e5"
Sep 08 13:15:41.048131 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:41.047979    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-br278"
Sep 08 13:15:41.048131 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:41.048075    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-br278" podUID="6fa64a88-f23c-42a6-8a1f-748e0278757a"
Sep 08 13:15:41.048807 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:41.048618    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:15:41.048807 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:41.048739    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5jt7" podUID="24a4cf01-77e6-44a0-8074-0929ee8dfee7"
Sep 08 13:15:41.077225 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:41.077185    2558 generic.go:356] "Generic (PLEG): container finished" podID="485a6b3955bdaff3eb439536885dfd93" containerID="b1773f2aa33fe3104fbdf2a60839ee127e69d41b5a4d8a4a6b95bf91125013db" exitCode=0
Sep 08 13:15:41.077407 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:41.077245    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-147.ec2.internal" event={"ID":"485a6b3955bdaff3eb439536885dfd93","Type":"ContainerDied","Data":"b1773f2aa33fe3104fbdf2a60839ee127e69d41b5a4d8a4a6b95bf91125013db"}
Sep 08 13:15:42.083144 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:42.082811    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-147.ec2.internal" event={"ID":"485a6b3955bdaff3eb439536885dfd93","Type":"ContainerStarted","Data":"f62d12427ec76ffbdb688612c00b9b65d68ffee100599c35b954d8ec20ddbffc"}
Sep 08 13:15:42.100811 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:42.100746    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-147.ec2.internal" podStartSLOduration=4.100723878 podStartE2EDuration="4.100723878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:15:42.100267301 +0000 UTC m=+5.552028239" watchObservedRunningTime="2026-09-08 13:15:42.100723878 +0000 UTC m=+5.552484814"
Sep 08 13:15:42.625129 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:42.625051    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90dadd3f-c403-4d64-af3a-45b4c68163e5-metrics-certs\") pod \"network-metrics-daemon-q4m28\" (UID: \"90dadd3f-c403-4d64-af3a-45b4c68163e5\") " pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:15:42.625326 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:42.625267    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:15:42.625396 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:42.625334    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90dadd3f-c403-4d64-af3a-45b4c68163e5-metrics-certs podName:90dadd3f-c403-4d64-af3a-45b4c68163e5 nodeName:}" failed. No retries permitted until 2026-09-08 13:15:46.625314346 +0000 UTC m=+10.077075280 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90dadd3f-c403-4d64-af3a-45b4c68163e5-metrics-certs") pod "network-metrics-daemon-q4m28" (UID: "90dadd3f-c403-4d64-af3a-45b4c68163e5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:15:42.725835 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:42.725783    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24a4cf01-77e6-44a0-8074-0929ee8dfee7-original-pull-secret\") pod \"global-pull-secret-syncer-g5jt7\" (UID: \"24a4cf01-77e6-44a0-8074-0929ee8dfee7\") " pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:15:42.726034 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:42.725972    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:15:42.726095 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:42.726039    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a4cf01-77e6-44a0-8074-0929ee8dfee7-original-pull-secret podName:24a4cf01-77e6-44a0-8074-0929ee8dfee7 nodeName:}" failed. No retries permitted until 2026-09-08 13:15:46.726020482 +0000 UTC m=+10.177781394 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24a4cf01-77e6-44a0-8074-0929ee8dfee7-original-pull-secret") pod "global-pull-secret-syncer-g5jt7" (UID: "24a4cf01-77e6-44a0-8074-0929ee8dfee7") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:15:42.827061 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:42.826957    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkd5f\" (UniqueName: \"kubernetes.io/projected/6fa64a88-f23c-42a6-8a1f-748e0278757a-kube-api-access-wkd5f\") pod \"network-check-target-br278\" (UID: \"6fa64a88-f23c-42a6-8a1f-748e0278757a\") " pod="openshift-network-diagnostics/network-check-target-br278"
Sep 08 13:15:42.827238 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:42.827146    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:15:42.827238 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:42.827174    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:15:42.827238 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:42.827189    2558 projected.go:196] Error preparing data for projected volume kube-api-access-wkd5f for pod openshift-network-diagnostics/network-check-target-br278: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:15:42.827434 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:42.827255    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fa64a88-f23c-42a6-8a1f-748e0278757a-kube-api-access-wkd5f podName:6fa64a88-f23c-42a6-8a1f-748e0278757a nodeName:}" failed. No retries permitted until 2026-09-08 13:15:46.827234518 +0000 UTC m=+10.278995429 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wkd5f" (UniqueName: "kubernetes.io/projected/6fa64a88-f23c-42a6-8a1f-748e0278757a-kube-api-access-wkd5f") pod "network-check-target-br278" (UID: "6fa64a88-f23c-42a6-8a1f-748e0278757a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:15:43.047948 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:43.047866    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-br278"
Sep 08 13:15:43.047948 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:43.047887    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:15:43.048129 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:43.047865    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:15:43.048129 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:43.048003    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-br278" podUID="6fa64a88-f23c-42a6-8a1f-748e0278757a"
Sep 08 13:15:43.048204 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:43.048128    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4m28" podUID="90dadd3f-c403-4d64-af3a-45b4c68163e5"
Sep 08 13:15:43.048204 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:43.048191    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5jt7" podUID="24a4cf01-77e6-44a0-8074-0929ee8dfee7"
Sep 08 13:15:45.047494 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:45.047432    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:15:45.048001 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:45.047562    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4m28" podUID="90dadd3f-c403-4d64-af3a-45b4c68163e5"
Sep 08 13:15:45.048001 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:45.047674    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-br278"
Sep 08 13:15:45.048001 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:45.047883    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-br278" podUID="6fa64a88-f23c-42a6-8a1f-748e0278757a"
Sep 08 13:15:45.048001 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:45.047930    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:15:45.048208 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:45.048006    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5jt7" podUID="24a4cf01-77e6-44a0-8074-0929ee8dfee7"
Sep 08 13:15:46.660217 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:46.660179    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90dadd3f-c403-4d64-af3a-45b4c68163e5-metrics-certs\") pod \"network-metrics-daemon-q4m28\" (UID: \"90dadd3f-c403-4d64-af3a-45b4c68163e5\") " pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:15:46.660685 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:46.660353    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:15:46.660685 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:46.660421    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90dadd3f-c403-4d64-af3a-45b4c68163e5-metrics-certs podName:90dadd3f-c403-4d64-af3a-45b4c68163e5 nodeName:}" failed. No retries permitted until 2026-09-08 13:15:54.660401515 +0000 UTC m=+18.112162443 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90dadd3f-c403-4d64-af3a-45b4c68163e5-metrics-certs") pod "network-metrics-daemon-q4m28" (UID: "90dadd3f-c403-4d64-af3a-45b4c68163e5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:15:46.761065 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:46.761020    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24a4cf01-77e6-44a0-8074-0929ee8dfee7-original-pull-secret\") pod \"global-pull-secret-syncer-g5jt7\" (UID: \"24a4cf01-77e6-44a0-8074-0929ee8dfee7\") " pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:15:46.761262 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:46.761194    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:15:46.761337 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:46.761286    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a4cf01-77e6-44a0-8074-0929ee8dfee7-original-pull-secret podName:24a4cf01-77e6-44a0-8074-0929ee8dfee7 nodeName:}" failed. No retries permitted until 2026-09-08 13:15:54.761264085 +0000 UTC m=+18.213025001 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24a4cf01-77e6-44a0-8074-0929ee8dfee7-original-pull-secret") pod "global-pull-secret-syncer-g5jt7" (UID: "24a4cf01-77e6-44a0-8074-0929ee8dfee7") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:15:46.861987 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:46.861949    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkd5f\" (UniqueName: \"kubernetes.io/projected/6fa64a88-f23c-42a6-8a1f-748e0278757a-kube-api-access-wkd5f\") pod \"network-check-target-br278\" (UID: \"6fa64a88-f23c-42a6-8a1f-748e0278757a\") " pod="openshift-network-diagnostics/network-check-target-br278"
Sep 08 13:15:46.862181 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:46.862126    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:15:46.862181 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:46.862153    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:15:46.862181 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:46.862168    2558 projected.go:196] Error preparing data for projected volume kube-api-access-wkd5f for pod openshift-network-diagnostics/network-check-target-br278: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:15:46.862341 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:46.862237    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fa64a88-f23c-42a6-8a1f-748e0278757a-kube-api-access-wkd5f podName:6fa64a88-f23c-42a6-8a1f-748e0278757a nodeName:}" failed. No retries permitted until 2026-09-08 13:15:54.86221799 +0000 UTC m=+18.313978900 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wkd5f" (UniqueName: "kubernetes.io/projected/6fa64a88-f23c-42a6-8a1f-748e0278757a-kube-api-access-wkd5f") pod "network-check-target-br278" (UID: "6fa64a88-f23c-42a6-8a1f-748e0278757a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:15:47.047177 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:47.047100    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:15:47.047177 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:47.047119    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-br278"
Sep 08 13:15:47.047177 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:47.047123    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:15:47.047440 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:47.047281    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-br278" podUID="6fa64a88-f23c-42a6-8a1f-748e0278757a"
Sep 08 13:15:47.047440 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:47.047303    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4m28" podUID="90dadd3f-c403-4d64-af3a-45b4c68163e5"
Sep 08 13:15:47.047440 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:47.047384    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5jt7" podUID="24a4cf01-77e6-44a0-8074-0929ee8dfee7"
Sep 08 13:15:49.047999 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:49.047950    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-br278"
Sep 08 13:15:49.047999 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:49.047957    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:15:49.048507 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:49.047990    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:15:49.048507 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:49.048057    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-br278" podUID="6fa64a88-f23c-42a6-8a1f-748e0278757a"
Sep 08 13:15:49.048507 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:49.048177    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4m28" podUID="90dadd3f-c403-4d64-af3a-45b4c68163e5"
Sep 08 13:15:49.048507 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:49.048272    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5jt7" podUID="24a4cf01-77e6-44a0-8074-0929ee8dfee7"
Sep 08 13:15:51.047652 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:51.047614    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:15:51.047652 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:51.047643    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:15:51.048167 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:51.047626    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-br278"
Sep 08 13:15:51.048167 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:51.047754    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4m28" podUID="90dadd3f-c403-4d64-af3a-45b4c68163e5"
Sep 08 13:15:51.048167 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:51.047832    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-br278" podUID="6fa64a88-f23c-42a6-8a1f-748e0278757a"
Sep 08 13:15:51.048167 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:51.047916    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5jt7" podUID="24a4cf01-77e6-44a0-8074-0929ee8dfee7"
Sep 08 13:15:53.047992 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:53.047952    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:15:53.048445 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:53.047952    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:15:53.048445 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:53.048073    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4m28" podUID="90dadd3f-c403-4d64-af3a-45b4c68163e5"
Sep 08 13:15:53.048445 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:53.047952    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-br278"
Sep 08 13:15:53.048445 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:53.048156    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5jt7" podUID="24a4cf01-77e6-44a0-8074-0929ee8dfee7"
Sep 08 13:15:53.048445 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:53.048200    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-br278" podUID="6fa64a88-f23c-42a6-8a1f-748e0278757a"
Sep 08 13:15:54.718130 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:54.718096    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90dadd3f-c403-4d64-af3a-45b4c68163e5-metrics-certs\") pod \"network-metrics-daemon-q4m28\" (UID: \"90dadd3f-c403-4d64-af3a-45b4c68163e5\") " pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:15:54.718626 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:54.718219    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:15:54.718626 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:54.718287    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90dadd3f-c403-4d64-af3a-45b4c68163e5-metrics-certs podName:90dadd3f-c403-4d64-af3a-45b4c68163e5 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:10.718269535 +0000 UTC m=+34.170030444 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90dadd3f-c403-4d64-af3a-45b4c68163e5-metrics-certs") pod "network-metrics-daemon-q4m28" (UID: "90dadd3f-c403-4d64-af3a-45b4c68163e5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:15:54.818466 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:54.818427    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24a4cf01-77e6-44a0-8074-0929ee8dfee7-original-pull-secret\") pod \"global-pull-secret-syncer-g5jt7\" (UID: \"24a4cf01-77e6-44a0-8074-0929ee8dfee7\") " pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:15:54.818658 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:54.818582    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:15:54.818718 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:54.818663    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a4cf01-77e6-44a0-8074-0929ee8dfee7-original-pull-secret podName:24a4cf01-77e6-44a0-8074-0929ee8dfee7 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:10.818646441 +0000 UTC m=+34.270407357 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24a4cf01-77e6-44a0-8074-0929ee8dfee7-original-pull-secret") pod "global-pull-secret-syncer-g5jt7" (UID: "24a4cf01-77e6-44a0-8074-0929ee8dfee7") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:15:54.919401 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:54.919365    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkd5f\" (UniqueName: \"kubernetes.io/projected/6fa64a88-f23c-42a6-8a1f-748e0278757a-kube-api-access-wkd5f\") pod \"network-check-target-br278\" (UID: \"6fa64a88-f23c-42a6-8a1f-748e0278757a\") " pod="openshift-network-diagnostics/network-check-target-br278"
Sep 08 13:15:54.919578 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:54.919517    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:15:54.919578 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:54.919534    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:15:54.919578 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:54.919544    2558 projected.go:196] Error preparing data for projected volume kube-api-access-wkd5f for pod openshift-network-diagnostics/network-check-target-br278: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:15:54.919738 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:54.919609    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fa64a88-f23c-42a6-8a1f-748e0278757a-kube-api-access-wkd5f podName:6fa64a88-f23c-42a6-8a1f-748e0278757a nodeName:}" failed. No retries permitted until 2026-09-08 13:16:10.919579844 +0000 UTC m=+34.371340753 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wkd5f" (UniqueName: "kubernetes.io/projected/6fa64a88-f23c-42a6-8a1f-748e0278757a-kube-api-access-wkd5f") pod "network-check-target-br278" (UID: "6fa64a88-f23c-42a6-8a1f-748e0278757a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:15:55.047176 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:55.047098    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:15:55.047176 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:55.047144    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-br278"
Sep 08 13:15:55.047390 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:55.047104    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:15:55.047390 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:55.047230    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5jt7" podUID="24a4cf01-77e6-44a0-8074-0929ee8dfee7"
Sep 08 13:15:55.047390 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:55.047312    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4m28" podUID="90dadd3f-c403-4d64-af3a-45b4c68163e5"
Sep 08 13:15:55.047390 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:55.047373    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-br278" podUID="6fa64a88-f23c-42a6-8a1f-748e0278757a"
Sep 08 13:15:56.965052 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:15:56.959107    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21e691d07f9ad0e45bf40e33342a7334.slice/crio-c4d7c49ab46d19bf6d6657d4b490471491da576ce9c7cda6cd14947d3ce0268c WatchSource:0}: Error finding container c4d7c49ab46d19bf6d6657d4b490471491da576ce9c7cda6cd14947d3ce0268c: Status 404 returned error can't find the container with id c4d7c49ab46d19bf6d6657d4b490471491da576ce9c7cda6cd14947d3ce0268c
Sep 08 13:15:57.050453 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:57.050424    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:15:57.050578 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:57.050554    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4m28" podUID="90dadd3f-c403-4d64-af3a-45b4c68163e5"
Sep 08 13:15:57.050895 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:57.050876    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:15:57.050997 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:57.050878    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-br278"
Sep 08 13:15:57.051072 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:57.050974    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5jt7" podUID="24a4cf01-77e6-44a0-8074-0929ee8dfee7"
Sep 08 13:15:57.051127 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:57.051088    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-br278" podUID="6fa64a88-f23c-42a6-8a1f-748e0278757a"
Sep 08 13:15:57.126083 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:57.125335    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-147.ec2.internal" event={"ID":"21e691d07f9ad0e45bf40e33342a7334","Type":"ContainerStarted","Data":"c4d7c49ab46d19bf6d6657d4b490471491da576ce9c7cda6cd14947d3ce0268c"}
Sep 08 13:15:57.132231 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:57.130869    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zss2s" event={"ID":"2913fac9-49a5-45a3-9181-b2d644239547","Type":"ContainerStarted","Data":"bf828ce417d26cb6ed68e6d39792143266ea527dbe35ccc7f36cb191105be732"}
Sep 08 13:15:57.138457 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:57.138428    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k7cdm" event={"ID":"d476dc00-0e12-46b0-900a-a95c96fd7ca3","Type":"ContainerStarted","Data":"6f4801e3995764415352a3e7f4d7cef736c2a73c122552d085257a6b3db2ad78"}
Sep 08 13:15:57.152859 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:57.152605    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zss2s" podStartSLOduration=2.8092923450000002 podStartE2EDuration="20.152567905s" totalImagesPullingTime="17.34327556s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:15:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:15:57.151314249 +0000 UTC m=+20.603075180" watchObservedRunningTime="2026-09-08 13:15:57.152567905 +0000 UTC m=+20.604328837"
Sep 08 13:15:57.170728 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:57.170685    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-k7cdm" podStartSLOduration=6.398772835 podStartE2EDuration="20.170668019s" totalImagesPullingTime="13.771895184s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:15:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:15:57.170204762 +0000 UTC m=+20.621965694" watchObservedRunningTime="2026-09-08 13:15:57.170668019 +0000 UTC m=+20.622428950"
Sep 08 13:15:58.142663 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:58.142429    2558 generic.go:356] "Generic (PLEG): container finished" podID="74540ccb-b9a8-424b-bae1-5d7cd5ce07fa" containerID="0d67f24fda0ae4c8e7e63f525bb37a45a4171136bedd8233c66690da834be2fe" exitCode=0
Sep 08 13:15:58.143526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:58.142480    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtgss" event={"ID":"74540ccb-b9a8-424b-bae1-5d7cd5ce07fa","Type":"ContainerDied","Data":"0d67f24fda0ae4c8e7e63f525bb37a45a4171136bedd8233c66690da834be2fe"}
Sep 08 13:15:58.144426 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:58.144369    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k9f94" event={"ID":"58dbf292-5174-4712-b380-d5763868b1c5","Type":"ContainerStarted","Data":"d5408f88003b98729146d4ccbf7029cfd1ddecdf80a9dd32d6ccc046517ee6cd"}
Sep 08 13:15:58.146032 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:58.146005    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vwvwg" event={"ID":"3ce2ec68-0166-484d-8e1a-28cc99334e5d","Type":"ContainerStarted","Data":"0cd919d195b059b484f0fcf5cc44fa895e18623e68438cd2d13e64d1ecf4529d"}
Sep 08 13:15:58.147910 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:58.147885    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7t44n" event={"ID":"eb60fb08-544c-4bd8-93fe-103f5f2bde69","Type":"ContainerStarted","Data":"0ef3b81b803f7fa8f01ea9e25e0221eded27a8c4307604937c15325dad06e3b0"}
Sep 08 13:15:58.151234 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:58.151213    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm" event={"ID":"97e67a4f-0d9f-4646-a1e5-2b12014a6242","Type":"ContainerStarted","Data":"cce79b1b46cf714828a5822e80e20e091964e046dbd5ee7d85c43f26b88ee2c6"}
Sep 08 13:15:58.151322 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:58.151290    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm" event={"ID":"97e67a4f-0d9f-4646-a1e5-2b12014a6242","Type":"ContainerStarted","Data":"4f62a8a0392a8b606ca82b7c1b0c9db208a791591875237001a2b65dea0dca25"}
Sep 08 13:15:58.151374 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:58.151337    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm" event={"ID":"97e67a4f-0d9f-4646-a1e5-2b12014a6242","Type":"ContainerStarted","Data":"85ec4a97c5317ff13c39479e1ced780a3894d1783563096ac068602cb7ba8033"}
Sep 08 13:15:58.151437 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:58.151377    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm" event={"ID":"97e67a4f-0d9f-4646-a1e5-2b12014a6242","Type":"ContainerStarted","Data":"36cc0db0821319b90f6a44527270a68332a42e6de1075eef22fcbd20383a8daf"}
Sep 08 13:15:58.151437 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:58.151415    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm" event={"ID":"97e67a4f-0d9f-4646-a1e5-2b12014a6242","Type":"ContainerStarted","Data":"dd093606b95fc5d06a5cf00f0d688ff8539957acb3db18a38d435df45291a8b9"}
Sep 08 13:15:58.151546 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:58.151447    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm" event={"ID":"97e67a4f-0d9f-4646-a1e5-2b12014a6242","Type":"ContainerStarted","Data":"c4fee14236527432f691b532d33f5d7483d4c063e235fdff8db825f48404c34b"}
Sep 08 13:15:58.152818 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:58.152757    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5zws" event={"ID":"eb97e09e-f5cf-48dd-a75c-f40e60d9756d","Type":"ContainerStarted","Data":"ccc1d2666516a0fdb9f4cc59811d2f29fc947c919d508a0212db32edf7205e72"}
Sep 08 13:15:58.181344 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:58.181293    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-k9f94" podStartSLOduration=3.880917771 podStartE2EDuration="21.181273591s" totalImagesPullingTime="17.30035582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:15:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:15:58.180657861 +0000 UTC m=+21.632418802" watchObservedRunningTime="2026-09-08 13:15:58.181273591 +0000 UTC m=+21.633034524"
Sep 08 13:15:58.195862 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:58.195811    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7t44n" podStartSLOduration=3.551788877 podStartE2EDuration="21.195791547s" totalImagesPullingTime="17.64400267s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:15:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:15:58.195438602 +0000 UTC m=+21.647199534" watchObservedRunningTime="2026-09-08 13:15:58.195791547 +0000 UTC m=+21.647552500"
Sep 08 13:15:58.358949 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:58.358918    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-k9f94"
Sep 08 13:15:58.359759 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:58.359738    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-k9f94"
Sep 08 13:15:58.375840 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:58.375796    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vwvwg" podStartSLOduration=4.082024291 podStartE2EDuration="21.375780186s" totalImagesPullingTime="17.293755895s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:15:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:15:58.21617186 +0000 UTC m=+21.667932792" watchObservedRunningTime="2026-09-08 13:15:58.375780186 +0000 UTC m=+21.827541116"
Sep 08 13:15:58.824307 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:58.824286    2558 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 08 13:15:59.048175 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:59.048098    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:15:59.048424 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:59.048098    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:15:59.048424 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:59.048206    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4m28" podUID="90dadd3f-c403-4d64-af3a-45b4c68163e5"
Sep 08 13:15:59.048424 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:59.048098    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-br278"
Sep 08 13:15:59.048424 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:59.048287    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5jt7" podUID="24a4cf01-77e6-44a0-8074-0929ee8dfee7"
Sep 08 13:15:59.048424 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:15:59.048378    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-br278" podUID="6fa64a88-f23c-42a6-8a1f-748e0278757a"
Sep 08 13:15:59.051881 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:59.051814    2558 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T13:15:58.824302954Z","UUID":"4bff5969-d675-469a-a9a8-a3a199328134","Handler":null,"Name":"","Endpoint":""}
Sep 08 13:15:59.053123 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:59.053110    2558 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 08 13:15:59.053217 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:59.053132    2558 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 08 13:15:59.156652 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:59.156617    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-147.ec2.internal" event={"ID":"21e691d07f9ad0e45bf40e33342a7334","Type":"ContainerStarted","Data":"17c3d14455abb0601fb72246eee9fbb391e6512b6da26ea93ce238d339e6fdca"}
Sep 08 13:15:59.158439 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:59.158413    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5zws" event={"ID":"eb97e09e-f5cf-48dd-a75c-f40e60d9756d","Type":"ContainerStarted","Data":"7b442fc5b667991516b294835204bf2562125f919ec3d0c69bdbacb77c4867ec"}
Sep 08 13:15:59.159799 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:59.159771    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-75fqq" event={"ID":"3bb8c2d9-5301-4487-a874-f2825ab4f458","Type":"ContainerStarted","Data":"a5a69fc0db2b64e34d2d66fb8617ebf9c24268c86254ec1c614fb729a20f9c8e"}
Sep 08 13:15:59.171025 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:59.170981    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-147.ec2.internal" podStartSLOduration=21.170968872 podStartE2EDuration="21.170968872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:15:59.170901997 +0000 UTC m=+22.622662927" watchObservedRunningTime="2026-09-08 13:15:59.170968872 +0000 UTC m=+22.622729803"
Sep 08 13:15:59.188641 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:15:59.188582    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-75fqq" podStartSLOduration=8.412587421 podStartE2EDuration="22.188568913s" totalImagesPullingTime="13.775981492s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:15:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:15:59.188366016 +0000 UTC m=+22.640126960" watchObservedRunningTime="2026-09-08 13:15:59.188568913 +0000 UTC m=+22.640329843"
Sep 08 13:16:00.164990 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:00.164951    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm" event={"ID":"97e67a4f-0d9f-4646-a1e5-2b12014a6242","Type":"ContainerStarted","Data":"95ff5a080e5760d5b9128f6df5d658cb9725b8b9b44cbc4dc42fb1316c4edbde"}
Sep 08 13:16:00.165363 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:00.164971    2558 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 08 13:16:00.521660 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:00.521558    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-k9f94"
Sep 08 13:16:00.522122 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:00.522105    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-k9f94"
Sep 08 13:16:01.047575 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:01.047537    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:16:01.047790 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:01.047537    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:16:01.047790 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:01.047604    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-br278"
Sep 08 13:16:01.047790 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:01.047678    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5jt7" podUID="24a4cf01-77e6-44a0-8074-0929ee8dfee7"
Sep 08 13:16:01.047790 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:01.047772    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4m28" podUID="90dadd3f-c403-4d64-af3a-45b4c68163e5"
Sep 08 13:16:01.048031 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:01.047839    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-br278" podUID="6fa64a88-f23c-42a6-8a1f-748e0278757a"
Sep 08 13:16:02.174029 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:02.173605    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm" event={"ID":"97e67a4f-0d9f-4646-a1e5-2b12014a6242","Type":"ContainerStarted","Data":"9b09a191a4929f8f70516a03997a772917f9c09c2ef192d5adbe206a752286cb"}
Sep 08 13:16:02.200416 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:02.200360    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm" podStartSLOduration=7.483717786 podStartE2EDuration="25.200342373s" totalImagesPullingTime="17.716624587s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:15:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:02.199726765 +0000 UTC m=+25.651487689" watchObservedRunningTime="2026-09-08 13:16:02.200342373 +0000 UTC m=+25.652103305"
Sep 08 13:16:03.051986 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:03.051958    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-br278"
Sep 08 13:16:03.052146 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:03.052091    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:16:03.052146 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:03.052107    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-br278" podUID="6fa64a88-f23c-42a6-8a1f-748e0278757a"
Sep 08 13:16:03.052250 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:03.052196    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5jt7" podUID="24a4cf01-77e6-44a0-8074-0929ee8dfee7"
Sep 08 13:16:03.052250 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:03.052244    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:16:03.052348 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:03.052323    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4m28" podUID="90dadd3f-c403-4d64-af3a-45b4c68163e5"
Sep 08 13:16:03.176913 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:03.176882    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:16:03.177478 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:03.176974    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:16:03.177478 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:03.177023    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:16:03.196693 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:03.196655    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:16:03.197227 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:03.197165    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:16:03.548910 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:03.548675    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g5jt7"]
Sep 08 13:16:03.548910 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:03.548848    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:16:03.549121 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:03.548959    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5jt7" podUID="24a4cf01-77e6-44a0-8074-0929ee8dfee7"
Sep 08 13:16:03.554636 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:03.554392    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-br278"]
Sep 08 13:16:03.554636 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:03.554518    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-br278"
Sep 08 13:16:03.554636 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:03.554570    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q4m28"]
Sep 08 13:16:03.554887 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:03.554632    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-br278" podUID="6fa64a88-f23c-42a6-8a1f-748e0278757a"
Sep 08 13:16:03.554887 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:03.554677    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:16:03.554887 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:03.554767    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4m28" podUID="90dadd3f-c403-4d64-af3a-45b4c68163e5"
Sep 08 13:16:04.181430 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:04.181385    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5zws" event={"ID":"eb97e09e-f5cf-48dd-a75c-f40e60d9756d","Type":"ContainerStarted","Data":"1137ac195248422996f0ef7396cf70faad3d50f0956bab8721a982a97904d95d"}
Sep 08 13:16:04.201721 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:04.201664    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m5zws" podStartSLOduration=7.190693416 podStartE2EDuration="27.201641743s" totalImagesPullingTime="20.010948327s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:15:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:04.20043954 +0000 UTC m=+27.652200483" watchObservedRunningTime="2026-09-08 13:16:04.201641743 +0000 UTC m=+27.653402672"
Sep 08 13:16:05.047465 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:05.047205    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:16:05.047656 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:05.047205    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-br278"
Sep 08 13:16:05.047656 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:05.047577    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5jt7" podUID="24a4cf01-77e6-44a0-8074-0929ee8dfee7"
Sep 08 13:16:05.047806 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:05.047667    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-br278" podUID="6fa64a88-f23c-42a6-8a1f-748e0278757a"
Sep 08 13:16:05.047806 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:05.047262    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:16:05.047903 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:05.047801    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4m28" podUID="90dadd3f-c403-4d64-af3a-45b4c68163e5"
Sep 08 13:16:06.189004 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:06.188958    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtgss" event={"ID":"74540ccb-b9a8-424b-bae1-5d7cd5ce07fa","Type":"ContainerStarted","Data":"1cb4c7fbf0122095a199cc7046c0f9cbde7f0df630efb63548af36cf306dcafc"}
Sep 08 13:16:07.048656 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:07.048618    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:16:07.048656 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:07.048659    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:16:07.048877 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:07.048745    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5jt7" podUID="24a4cf01-77e6-44a0-8074-0929ee8dfee7"
Sep 08 13:16:07.048877 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:07.048785    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4m28" podUID="90dadd3f-c403-4d64-af3a-45b4c68163e5"
Sep 08 13:16:07.048877 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:07.048856    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-br278"
Sep 08 13:16:07.048986 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:07.048955    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-br278" podUID="6fa64a88-f23c-42a6-8a1f-748e0278757a"
Sep 08 13:16:07.193014 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:07.192978    2558 generic.go:356] "Generic (PLEG): container finished" podID="74540ccb-b9a8-424b-bae1-5d7cd5ce07fa" containerID="1cb4c7fbf0122095a199cc7046c0f9cbde7f0df630efb63548af36cf306dcafc" exitCode=0
Sep 08 13:16:07.193497 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:07.193033    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtgss" event={"ID":"74540ccb-b9a8-424b-bae1-5d7cd5ce07fa","Type":"ContainerDied","Data":"1cb4c7fbf0122095a199cc7046c0f9cbde7f0df630efb63548af36cf306dcafc"}
Sep 08 13:16:09.047259 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:09.047225    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:16:09.047732 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:09.047339    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g5jt7" podUID="24a4cf01-77e6-44a0-8074-0929ee8dfee7"
Sep 08 13:16:09.047732 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:09.047235    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:16:09.047732 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:09.047441    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4m28" podUID="90dadd3f-c403-4d64-af3a-45b4c68163e5"
Sep 08 13:16:09.047732 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:09.047225    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-br278"
Sep 08 13:16:09.047732 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:09.047609    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-br278" podUID="6fa64a88-f23c-42a6-8a1f-748e0278757a"
Sep 08 13:16:09.199469 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:09.199437    2558 generic.go:356] "Generic (PLEG): container finished" podID="74540ccb-b9a8-424b-bae1-5d7cd5ce07fa" containerID="b03dc93e53919e08d092e555e18f8872464e3f2b04de6d5dbc4ff2753c5f8868" exitCode=0
Sep 08 13:16:09.199677 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:09.199492    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtgss" event={"ID":"74540ccb-b9a8-424b-bae1-5d7cd5ce07fa","Type":"ContainerDied","Data":"b03dc93e53919e08d092e555e18f8872464e3f2b04de6d5dbc4ff2753c5f8868"}
Sep 08 13:16:09.867780 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:09.867580    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-147.ec2.internal" event="NodeReady"
Sep 08 13:16:09.867939 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:09.867881    2558 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 08 13:16:09.909364 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:09.909331    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-bcb89cbc9-g9vr7"]
Sep 08 13:16:09.925511 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:09.924798    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:16:09.925511 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:09.925207    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bcb89cbc9-g9vr7"]
Sep 08 13:16:09.926893 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:09.926868    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 08 13:16:09.927009 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:09.926991    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 08 13:16:09.927171 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:09.927149    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wqhr2\""
Sep 08 13:16:09.927496 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:09.927461    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 08 13:16:09.934222 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:09.934194    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 08 13:16:09.941255 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:09.941231    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pvnnp"]
Sep 08 13:16:09.956573 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:09.956537    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pvnnp"]
Sep 08 13:16:09.956757 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:09.956681    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pvnnp"
Sep 08 13:16:09.958990 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:09.958968    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 08 13:16:09.959451 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:09.959430    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8shlb\""
Sep 08 13:16:09.959451 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:09.959443    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 08 13:16:10.023326 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.023291    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frtr6\" (UniqueName: \"kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-kube-api-access-frtr6\") pod \"image-registry-bcb89cbc9-g9vr7\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") " pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:16:10.023326 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.023329    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7bcc0442-53c5-4088-a419-10e4af66a483-config-volume\") pod \"dns-default-pvnnp\" (UID: \"7bcc0442-53c5-4088-a419-10e4af66a483\") " pod="openshift-dns/dns-default-pvnnp"
Sep 08 13:16:10.023609 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.023366    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-ca-trust-extracted\") pod \"image-registry-bcb89cbc9-g9vr7\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") " pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:16:10.023609 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.023391    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-certificates\") pod \"image-registry-bcb89cbc9-g9vr7\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") " pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:16:10.023609 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.023406    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-installation-pull-secrets\") pod \"image-registry-bcb89cbc9-g9vr7\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") " pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:16:10.023609 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.023425    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-trusted-ca\") pod \"image-registry-bcb89cbc9-g9vr7\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") " pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:16:10.023609 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.023454    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bcc0442-53c5-4088-a419-10e4af66a483-metrics-tls\") pod \"dns-default-pvnnp\" (UID: \"7bcc0442-53c5-4088-a419-10e4af66a483\") " pod="openshift-dns/dns-default-pvnnp"
Sep 08 13:16:10.023609 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.023485    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-tls\") pod \"image-registry-bcb89cbc9-g9vr7\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") " pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:16:10.023609 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.023527    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bcc0442-53c5-4088-a419-10e4af66a483-tmp-dir\") pod \"dns-default-pvnnp\" (UID: \"7bcc0442-53c5-4088-a419-10e4af66a483\") " pod="openshift-dns/dns-default-pvnnp"
Sep 08 13:16:10.023609 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.023550    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kr4g\" (UniqueName: \"kubernetes.io/projected/7bcc0442-53c5-4088-a419-10e4af66a483-kube-api-access-6kr4g\") pod \"dns-default-pvnnp\" (UID: \"7bcc0442-53c5-4088-a419-10e4af66a483\") " pod="openshift-dns/dns-default-pvnnp"
Sep 08 13:16:10.023609 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.023566    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-image-registry-private-configuration\") pod \"image-registry-bcb89cbc9-g9vr7\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") " pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:16:10.023609 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.023583    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-bound-sa-token\") pod \"image-registry-bcb89cbc9-g9vr7\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") " pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:16:10.049495 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.049463    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-d2mvr"]
Sep 08 13:16:10.066771 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.066739    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d2mvr"]
Sep 08 13:16:10.066914 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.066835    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d2mvr"
Sep 08 13:16:10.069063 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.069039    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 08 13:16:10.069367 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.069352    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 08 13:16:10.069441 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.069425    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 08 13:16:10.069484 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.069442    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8cl4h\""
Sep 08 13:16:10.124958 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.124873    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kr4g\" (UniqueName: \"kubernetes.io/projected/7bcc0442-53c5-4088-a419-10e4af66a483-kube-api-access-6kr4g\") pod \"dns-default-pvnnp\" (UID: \"7bcc0442-53c5-4088-a419-10e4af66a483\") " pod="openshift-dns/dns-default-pvnnp"
Sep 08 13:16:10.124958 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.124914    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-image-registry-private-configuration\") pod \"image-registry-bcb89cbc9-g9vr7\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") " pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:16:10.124958 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.124934    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-bound-sa-token\") pod \"image-registry-bcb89cbc9-g9vr7\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") " pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:16:10.125232 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.125113    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frtr6\" (UniqueName: \"kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-kube-api-access-frtr6\") pod \"image-registry-bcb89cbc9-g9vr7\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") " pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:16:10.125232 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.125145    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7bcc0442-53c5-4088-a419-10e4af66a483-config-volume\") pod \"dns-default-pvnnp\" (UID: \"7bcc0442-53c5-4088-a419-10e4af66a483\") " pod="openshift-dns/dns-default-pvnnp"
Sep 08 13:16:10.125232 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.125180    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-ca-trust-extracted\") pod \"image-registry-bcb89cbc9-g9vr7\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") " pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:16:10.125232 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.125205    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-certificates\") pod \"image-registry-bcb89cbc9-g9vr7\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") " pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:16:10.125422 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.125321    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-installation-pull-secrets\") pod \"image-registry-bcb89cbc9-g9vr7\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") " pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:16:10.125422 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.125354    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/349aa96b-eaec-45cf-9a20-7f2227273c2c-cert\") pod \"ingress-canary-d2mvr\" (UID: \"349aa96b-eaec-45cf-9a20-7f2227273c2c\") " pod="openshift-ingress-canary/ingress-canary-d2mvr"
Sep 08 13:16:10.125422 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.125385    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-trusted-ca\") pod \"image-registry-bcb89cbc9-g9vr7\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") " pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:16:10.125568 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.125428    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bcc0442-53c5-4088-a419-10e4af66a483-metrics-tls\") pod \"dns-default-pvnnp\" (UID: \"7bcc0442-53c5-4088-a419-10e4af66a483\") " pod="openshift-dns/dns-default-pvnnp"
Sep 08 13:16:10.125568 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.125453    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-tls\") pod \"image-registry-bcb89cbc9-g9vr7\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") " pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:16:10.125568 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.125493    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bcc0442-53c5-4088-a419-10e4af66a483-tmp-dir\") pod \"dns-default-pvnnp\" (UID: \"7bcc0442-53c5-4088-a419-10e4af66a483\") " pod="openshift-dns/dns-default-pvnnp"
Sep 08 13:16:10.125568 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.125527    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h2fk\" (UniqueName: \"kubernetes.io/projected/349aa96b-eaec-45cf-9a20-7f2227273c2c-kube-api-access-4h2fk\") pod \"ingress-canary-d2mvr\" (UID: \"349aa96b-eaec-45cf-9a20-7f2227273c2c\") " pod="openshift-ingress-canary/ingress-canary-d2mvr"
Sep 08 13:16:10.125568 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:10.125543    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:16:10.125840 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:10.125632    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bcc0442-53c5-4088-a419-10e4af66a483-metrics-tls podName:7bcc0442-53c5-4088-a419-10e4af66a483 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:10.625584387 +0000 UTC m=+34.077345309 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7bcc0442-53c5-4088-a419-10e4af66a483-metrics-tls") pod "dns-default-pvnnp" (UID: "7bcc0442-53c5-4088-a419-10e4af66a483") : secret "dns-default-metrics-tls" not found
Sep 08 13:16:10.125840 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.125709    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-ca-trust-extracted\") pod \"image-registry-bcb89cbc9-g9vr7\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") " pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:16:10.125840 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:10.125730    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:16:10.125840 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:10.125744    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bcb89cbc9-g9vr7: secret "image-registry-tls" not found
Sep 08 13:16:10.125840 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:10.125787    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-tls podName:494d9fb4-e6b1-4d95-a27d-194c4d4c6079 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:10.625772424 +0000 UTC m=+34.077533347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-tls") pod "image-registry-bcb89cbc9-g9vr7" (UID: "494d9fb4-e6b1-4d95-a27d-194c4d4c6079") : secret "image-registry-tls" not found
Sep 08 13:16:10.125840 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.125833    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7bcc0442-53c5-4088-a419-10e4af66a483-config-volume\") pod \"dns-default-pvnnp\" (UID: \"7bcc0442-53c5-4088-a419-10e4af66a483\") " pod="openshift-dns/dns-default-pvnnp"
Sep 08 13:16:10.126042 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.125875    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-certificates\") pod \"image-registry-bcb89cbc9-g9vr7\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") " pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:16:10.126360 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.126338    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-trusted-ca\") pod \"image-registry-bcb89cbc9-g9vr7\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") " pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:16:10.129226 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.129202    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-installation-pull-secrets\") pod \"image-registry-bcb89cbc9-g9vr7\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") " pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:16:10.129322 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.129206    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-image-registry-private-configuration\") pod \"image-registry-bcb89cbc9-g9vr7\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") " pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:16:10.132576 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.132555    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bcc0442-53c5-4088-a419-10e4af66a483-tmp-dir\") pod \"dns-default-pvnnp\" (UID: \"7bcc0442-53c5-4088-a419-10e4af66a483\") " pod="openshift-dns/dns-default-pvnnp"
Sep 08 13:16:10.133785 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.133764    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-bound-sa-token\") pod \"image-registry-bcb89cbc9-g9vr7\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") " pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:16:10.134005 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.133975    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kr4g\" (UniqueName: \"kubernetes.io/projected/7bcc0442-53c5-4088-a419-10e4af66a483-kube-api-access-6kr4g\") pod \"dns-default-pvnnp\" (UID: \"7bcc0442-53c5-4088-a419-10e4af66a483\") " pod="openshift-dns/dns-default-pvnnp"
Sep 08 13:16:10.135262 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.135240    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frtr6\" (UniqueName: \"kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-kube-api-access-frtr6\") pod \"image-registry-bcb89cbc9-g9vr7\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") " pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:16:10.225957 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.225924    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/349aa96b-eaec-45cf-9a20-7f2227273c2c-cert\") pod \"ingress-canary-d2mvr\" (UID: \"349aa96b-eaec-45cf-9a20-7f2227273c2c\") " pod="openshift-ingress-canary/ingress-canary-d2mvr"
Sep 08 13:16:10.226160 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.225982    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h2fk\" (UniqueName: \"kubernetes.io/projected/349aa96b-eaec-45cf-9a20-7f2227273c2c-kube-api-access-4h2fk\") pod \"ingress-canary-d2mvr\" (UID: \"349aa96b-eaec-45cf-9a20-7f2227273c2c\") " pod="openshift-ingress-canary/ingress-canary-d2mvr"
Sep 08 13:16:10.228296 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.228266    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/349aa96b-eaec-45cf-9a20-7f2227273c2c-cert\") pod \"ingress-canary-d2mvr\" (UID: \"349aa96b-eaec-45cf-9a20-7f2227273c2c\") " pod="openshift-ingress-canary/ingress-canary-d2mvr"
Sep 08 13:16:10.233536 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.233510    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h2fk\" (UniqueName: \"kubernetes.io/projected/349aa96b-eaec-45cf-9a20-7f2227273c2c-kube-api-access-4h2fk\") pod \"ingress-canary-d2mvr\" (UID: \"349aa96b-eaec-45cf-9a20-7f2227273c2c\") " pod="openshift-ingress-canary/ingress-canary-d2mvr"
Sep 08 13:16:10.375102 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.375041    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d2mvr"
Sep 08 13:16:10.630008 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.629906    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bcc0442-53c5-4088-a419-10e4af66a483-metrics-tls\") pod \"dns-default-pvnnp\" (UID: \"7bcc0442-53c5-4088-a419-10e4af66a483\") " pod="openshift-dns/dns-default-pvnnp"
Sep 08 13:16:10.630008 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.629954    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-tls\") pod \"image-registry-bcb89cbc9-g9vr7\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") " pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:16:10.630229 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:10.630070    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:16:10.630229 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:10.630104    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:16:10.630229 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:10.630118    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bcb89cbc9-g9vr7: secret "image-registry-tls" not found
Sep 08 13:16:10.630229 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:10.630158    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bcc0442-53c5-4088-a419-10e4af66a483-metrics-tls podName:7bcc0442-53c5-4088-a419-10e4af66a483 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:11.630132998 +0000 UTC m=+35.081893926 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7bcc0442-53c5-4088-a419-10e4af66a483-metrics-tls") pod "dns-default-pvnnp" (UID: "7bcc0442-53c5-4088-a419-10e4af66a483") : secret "dns-default-metrics-tls" not found
Sep 08 13:16:10.630229 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:10.630174    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-tls podName:494d9fb4-e6b1-4d95-a27d-194c4d4c6079 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:11.630167636 +0000 UTC m=+35.081928546 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-tls") pod "image-registry-bcb89cbc9-g9vr7" (UID: "494d9fb4-e6b1-4d95-a27d-194c4d4c6079") : secret "image-registry-tls" not found
Sep 08 13:16:10.692439 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.692410    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d2mvr"]
Sep 08 13:16:10.695826 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:16:10.695793    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod349aa96b_eaec_45cf_9a20_7f2227273c2c.slice/crio-c76040fe38999300406315f41675f38c8ebf95d76bb089cea937cb45f383bbef WatchSource:0}: Error finding container c76040fe38999300406315f41675f38c8ebf95d76bb089cea937cb45f383bbef: Status 404 returned error can't find the container with id c76040fe38999300406315f41675f38c8ebf95d76bb089cea937cb45f383bbef
Sep 08 13:16:10.730354 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.730319    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90dadd3f-c403-4d64-af3a-45b4c68163e5-metrics-certs\") pod \"network-metrics-daemon-q4m28\" (UID: \"90dadd3f-c403-4d64-af3a-45b4c68163e5\") " pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:16:10.730517 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:10.730473    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:10.730562 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:10.730551    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90dadd3f-c403-4d64-af3a-45b4c68163e5-metrics-certs podName:90dadd3f-c403-4d64-af3a-45b4c68163e5 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:42.730534104 +0000 UTC m=+66.182295027 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90dadd3f-c403-4d64-af3a-45b4c68163e5-metrics-certs") pod "network-metrics-daemon-q4m28" (UID: "90dadd3f-c403-4d64-af3a-45b4c68163e5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:10.816033 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.815991    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k9f94_58dbf292-5174-4712-b380-d5763868b1c5/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:10.830856 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.830817    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24a4cf01-77e6-44a0-8074-0929ee8dfee7-original-pull-secret\") pod \"global-pull-secret-syncer-g5jt7\" (UID: \"24a4cf01-77e6-44a0-8074-0929ee8dfee7\") " pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:16:10.831025 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:10.830987    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:10.831092 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:10.831062    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a4cf01-77e6-44a0-8074-0929ee8dfee7-original-pull-secret podName:24a4cf01-77e6-44a0-8074-0929ee8dfee7 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:42.83104243 +0000 UTC m=+66.282803354 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/24a4cf01-77e6-44a0-8074-0929ee8dfee7-original-pull-secret") pod "global-pull-secret-syncer-g5jt7" (UID: "24a4cf01-77e6-44a0-8074-0929ee8dfee7") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:10.920248 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.920164    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k9f94_58dbf292-5174-4712-b380-d5763868b1c5/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:10.932171 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:10.932138    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkd5f\" (UniqueName: \"kubernetes.io/projected/6fa64a88-f23c-42a6-8a1f-748e0278757a-kube-api-access-wkd5f\") pod \"network-check-target-br278\" (UID: \"6fa64a88-f23c-42a6-8a1f-748e0278757a\") " pod="openshift-network-diagnostics/network-check-target-br278"
Sep 08 13:16:10.932311 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:10.932295    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:10.932351 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:10.932315    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:10.932351 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:10.932326    2558 projected.go:196] Error preparing data for projected volume kube-api-access-wkd5f for pod openshift-network-diagnostics/network-check-target-br278: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:10.932416 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:10.932373    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fa64a88-f23c-42a6-8a1f-748e0278757a-kube-api-access-wkd5f podName:6fa64a88-f23c-42a6-8a1f-748e0278757a nodeName:}" failed. No retries permitted until 2026-09-08 13:16:42.932360183 +0000 UTC m=+66.384121091 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wkd5f" (UniqueName: "kubernetes.io/projected/6fa64a88-f23c-42a6-8a1f-748e0278757a-kube-api-access-wkd5f") pod "network-check-target-br278" (UID: "6fa64a88-f23c-42a6-8a1f-748e0278757a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:11.028604 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:11.028547    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k9f94_58dbf292-5174-4712-b380-d5763868b1c5/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:11.047780 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:11.047749    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-br278"
Sep 08 13:16:11.047780 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:11.047751    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:16:11.047983 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:11.047871    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:16:11.050653 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:11.050471    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 13:16:11.050653 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:11.050484    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 13:16:11.050653 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:11.050476    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 13:16:11.050653 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:11.050528    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 13:16:11.051413 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:11.050670    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-x5pb6\""
Sep 08 13:16:11.051413 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:11.050816    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4rwvw\""
Sep 08 13:16:11.105437 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:11.105399    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-k9f94_58dbf292-5174-4712-b380-d5763868b1c5/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:11.205320 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:11.205226    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d2mvr" event={"ID":"349aa96b-eaec-45cf-9a20-7f2227273c2c","Type":"ContainerStarted","Data":"c76040fe38999300406315f41675f38c8ebf95d76bb089cea937cb45f383bbef"}
Sep 08 13:16:11.207714 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:11.207671    2558 generic.go:356] "Generic (PLEG): container finished" podID="74540ccb-b9a8-424b-bae1-5d7cd5ce07fa" containerID="323cfd967cea939ddd65ad816920388f32077e133504c6af568a2f7c0afe0c76" exitCode=0
Sep 08 13:16:11.207845 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:11.207726    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtgss" event={"ID":"74540ccb-b9a8-424b-bae1-5d7cd5ce07fa","Type":"ContainerDied","Data":"323cfd967cea939ddd65ad816920388f32077e133504c6af568a2f7c0afe0c76"}
Sep 08 13:16:11.639484 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:11.639444    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bcc0442-53c5-4088-a419-10e4af66a483-metrics-tls\") pod \"dns-default-pvnnp\" (UID: \"7bcc0442-53c5-4088-a419-10e4af66a483\") " pod="openshift-dns/dns-default-pvnnp"
Sep 08 13:16:11.639484 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:11.639484    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-tls\") pod \"image-registry-bcb89cbc9-g9vr7\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") " pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:16:11.639705 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:11.639619    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:16:11.639705 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:11.639685    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:16:11.639705 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:11.639703    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bcb89cbc9-g9vr7: secret "image-registry-tls" not found
Sep 08 13:16:11.639705 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:11.639705    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bcc0442-53c5-4088-a419-10e4af66a483-metrics-tls podName:7bcc0442-53c5-4088-a419-10e4af66a483 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:13.639688158 +0000 UTC m=+37.091449066 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7bcc0442-53c5-4088-a419-10e4af66a483-metrics-tls") pod "dns-default-pvnnp" (UID: "7bcc0442-53c5-4088-a419-10e4af66a483") : secret "dns-default-metrics-tls" not found
Sep 08 13:16:11.639835 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:11.639753    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-tls podName:494d9fb4-e6b1-4d95-a27d-194c4d4c6079 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:13.63974171 +0000 UTC m=+37.091502619 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-tls") pod "image-registry-bcb89cbc9-g9vr7" (UID: "494d9fb4-e6b1-4d95-a27d-194c4d4c6079") : secret "image-registry-tls" not found
Sep 08 13:16:13.654964 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:13.654763    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bcc0442-53c5-4088-a419-10e4af66a483-metrics-tls\") pod \"dns-default-pvnnp\" (UID: \"7bcc0442-53c5-4088-a419-10e4af66a483\") " pod="openshift-dns/dns-default-pvnnp"
Sep 08 13:16:13.654964 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:13.654951    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:16:13.655481 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:13.654979    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-tls\") pod \"image-registry-bcb89cbc9-g9vr7\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") " pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:16:13.655481 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:13.655025    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bcc0442-53c5-4088-a419-10e4af66a483-metrics-tls podName:7bcc0442-53c5-4088-a419-10e4af66a483 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:17.65500175 +0000 UTC m=+41.106762664 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7bcc0442-53c5-4088-a419-10e4af66a483-metrics-tls") pod "dns-default-pvnnp" (UID: "7bcc0442-53c5-4088-a419-10e4af66a483") : secret "dns-default-metrics-tls" not found
Sep 08 13:16:13.655481 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:13.655091    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:16:13.655481 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:13.655106    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bcb89cbc9-g9vr7: secret "image-registry-tls" not found
Sep 08 13:16:13.655481 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:13.655168    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-tls podName:494d9fb4-e6b1-4d95-a27d-194c4d4c6079 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:17.655152793 +0000 UTC m=+41.106913717 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-tls") pod "image-registry-bcb89cbc9-g9vr7" (UID: "494d9fb4-e6b1-4d95-a27d-194c4d4c6079") : secret "image-registry-tls" not found
Sep 08 13:16:17.689445 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:17.689342    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bcc0442-53c5-4088-a419-10e4af66a483-metrics-tls\") pod \"dns-default-pvnnp\" (UID: \"7bcc0442-53c5-4088-a419-10e4af66a483\") " pod="openshift-dns/dns-default-pvnnp"
Sep 08 13:16:17.689445 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:17.689384    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-tls\") pod \"image-registry-bcb89cbc9-g9vr7\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") " pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:16:17.689937 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:17.689500    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:16:17.689937 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:17.689511    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bcb89cbc9-g9vr7: secret "image-registry-tls" not found
Sep 08 13:16:17.689937 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:17.689516    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:16:17.689937 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:17.689560    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-tls podName:494d9fb4-e6b1-4d95-a27d-194c4d4c6079 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:25.689548312 +0000 UTC m=+49.141309222 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-tls") pod "image-registry-bcb89cbc9-g9vr7" (UID: "494d9fb4-e6b1-4d95-a27d-194c4d4c6079") : secret "image-registry-tls" not found
Sep 08 13:16:17.689937 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:17.689574    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bcc0442-53c5-4088-a419-10e4af66a483-metrics-tls podName:7bcc0442-53c5-4088-a419-10e4af66a483 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:25.689568501 +0000 UTC m=+49.141329410 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7bcc0442-53c5-4088-a419-10e4af66a483-metrics-tls") pod "dns-default-pvnnp" (UID: "7bcc0442-53c5-4088-a419-10e4af66a483") : secret "dns-default-metrics-tls" not found
Sep 08 13:16:18.226829 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:18.226789    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d2mvr" event={"ID":"349aa96b-eaec-45cf-9a20-7f2227273c2c","Type":"ContainerStarted","Data":"3ea8fb25ce9f97d22487f15bb5031e91da927fc081c7c534b2d696ffbbffb52b"}
Sep 08 13:16:18.229643 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:18.229610    2558 generic.go:356] "Generic (PLEG): container finished" podID="74540ccb-b9a8-424b-bae1-5d7cd5ce07fa" containerID="d006384030d362576c56e68c0c6019e1487de0872476f6c2fbe4058d69dee4ba" exitCode=0
Sep 08 13:16:18.229761 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:18.229670    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtgss" event={"ID":"74540ccb-b9a8-424b-bae1-5d7cd5ce07fa","Type":"ContainerDied","Data":"d006384030d362576c56e68c0c6019e1487de0872476f6c2fbe4058d69dee4ba"}
Sep 08 13:16:18.246184 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:18.246139    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-d2mvr" podStartSLOduration=1.795977331 podStartE2EDuration="8.246121476s" totalImagesPullingTime="6.450144145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:18.245425627 +0000 UTC m=+41.697186559" watchObservedRunningTime="2026-09-08 13:16:18.246121476 +0000 UTC m=+41.697882432"
Sep 08 13:16:19.234477 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:19.234381    2558 generic.go:356] "Generic (PLEG): container finished" podID="74540ccb-b9a8-424b-bae1-5d7cd5ce07fa" containerID="b43d03e5519ac0a7d5b80b6e49d95ca871f3a4be5bc4ed078d9cfb3ac2d0f9a3" exitCode=0
Sep 08 13:16:19.234897 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:19.234470    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtgss" event={"ID":"74540ccb-b9a8-424b-bae1-5d7cd5ce07fa","Type":"ContainerDied","Data":"b43d03e5519ac0a7d5b80b6e49d95ca871f3a4be5bc4ed078d9cfb3ac2d0f9a3"}
Sep 08 13:16:20.239911 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:20.239871    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtgss" event={"ID":"74540ccb-b9a8-424b-bae1-5d7cd5ce07fa","Type":"ContainerStarted","Data":"dbbb7ec69530b31f05e43698045782f4a660092b978087e3da4a7f1f77ae525a"}
Sep 08 13:16:20.262951 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:20.262900    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jtgss" podStartSLOduration=9.831748661 podStartE2EDuration="43.26288335s" totalImagesPullingTime="33.431134689s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:15:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:20.261991799 +0000 UTC m=+43.713752730" watchObservedRunningTime="2026-09-08 13:16:20.26288335 +0000 UTC m=+43.714644280"
Sep 08 13:16:25.751098 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:25.751049    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bcc0442-53c5-4088-a419-10e4af66a483-metrics-tls\") pod \"dns-default-pvnnp\" (UID: \"7bcc0442-53c5-4088-a419-10e4af66a483\") " pod="openshift-dns/dns-default-pvnnp"
Sep 08 13:16:25.751098 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:25.751101    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-tls\") pod \"image-registry-bcb89cbc9-g9vr7\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") " pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:16:25.751553 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:25.751214    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:16:25.751553 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:25.751270    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:16:25.751553 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:25.751291    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bcb89cbc9-g9vr7: secret "image-registry-tls" not found
Sep 08 13:16:25.751553 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:25.751275    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bcc0442-53c5-4088-a419-10e4af66a483-metrics-tls podName:7bcc0442-53c5-4088-a419-10e4af66a483 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:41.751259902 +0000 UTC m=+65.203020811 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7bcc0442-53c5-4088-a419-10e4af66a483-metrics-tls") pod "dns-default-pvnnp" (UID: "7bcc0442-53c5-4088-a419-10e4af66a483") : secret "dns-default-metrics-tls" not found
Sep 08 13:16:25.751553 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:25.751355    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-tls podName:494d9fb4-e6b1-4d95-a27d-194c4d4c6079 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:41.751341804 +0000 UTC m=+65.203102714 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-tls") pod "image-registry-bcb89cbc9-g9vr7" (UID: "494d9fb4-e6b1-4d95-a27d-194c4d4c6079") : secret "image-registry-tls" not found
Sep 08 13:16:35.194489 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:35.194446    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tzwrm"
Sep 08 13:16:41.764293 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:41.764234    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bcc0442-53c5-4088-a419-10e4af66a483-metrics-tls\") pod \"dns-default-pvnnp\" (UID: \"7bcc0442-53c5-4088-a419-10e4af66a483\") " pod="openshift-dns/dns-default-pvnnp"
Sep 08 13:16:41.764293 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:41.764289    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-tls\") pod \"image-registry-bcb89cbc9-g9vr7\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") " pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:16:41.764744 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:41.764395    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:16:41.764744 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:41.764459    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bcc0442-53c5-4088-a419-10e4af66a483-metrics-tls podName:7bcc0442-53c5-4088-a419-10e4af66a483 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:13.764443008 +0000 UTC m=+97.216203935 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7bcc0442-53c5-4088-a419-10e4af66a483-metrics-tls") pod "dns-default-pvnnp" (UID: "7bcc0442-53c5-4088-a419-10e4af66a483") : secret "dns-default-metrics-tls" not found
Sep 08 13:16:41.764744 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:41.764400    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:16:41.764744 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:41.764479    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bcb89cbc9-g9vr7: secret "image-registry-tls" not found
Sep 08 13:16:41.764744 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:41.764517    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-tls podName:494d9fb4-e6b1-4d95-a27d-194c4d4c6079 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:13.764507828 +0000 UTC m=+97.216268751 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-tls") pod "image-registry-bcb89cbc9-g9vr7" (UID: "494d9fb4-e6b1-4d95-a27d-194c4d4c6079") : secret "image-registry-tls" not found
Sep 08 13:16:42.772482 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:42.772440    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90dadd3f-c403-4d64-af3a-45b4c68163e5-metrics-certs\") pod \"network-metrics-daemon-q4m28\" (UID: \"90dadd3f-c403-4d64-af3a-45b4c68163e5\") " pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:16:42.774826 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:42.774800    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 13:16:42.782771 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:42.782750    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 08 13:16:42.782842 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:16:42.782831    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90dadd3f-c403-4d64-af3a-45b4c68163e5-metrics-certs podName:90dadd3f-c403-4d64-af3a-45b4c68163e5 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:46.782813376 +0000 UTC m=+130.234574289 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90dadd3f-c403-4d64-af3a-45b4c68163e5-metrics-certs") pod "network-metrics-daemon-q4m28" (UID: "90dadd3f-c403-4d64-af3a-45b4c68163e5") : secret "metrics-daemon-secret" not found
Sep 08 13:16:42.873347 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:42.873314    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24a4cf01-77e6-44a0-8074-0929ee8dfee7-original-pull-secret\") pod \"global-pull-secret-syncer-g5jt7\" (UID: \"24a4cf01-77e6-44a0-8074-0929ee8dfee7\") " pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:16:42.875816 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:42.875797    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 13:16:42.886263 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:42.886231    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/24a4cf01-77e6-44a0-8074-0929ee8dfee7-original-pull-secret\") pod \"global-pull-secret-syncer-g5jt7\" (UID: \"24a4cf01-77e6-44a0-8074-0929ee8dfee7\") " pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:16:42.973959 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:42.973924    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkd5f\" (UniqueName: \"kubernetes.io/projected/6fa64a88-f23c-42a6-8a1f-748e0278757a-kube-api-access-wkd5f\") pod \"network-check-target-br278\" (UID: \"6fa64a88-f23c-42a6-8a1f-748e0278757a\") " pod="openshift-network-diagnostics/network-check-target-br278"
Sep 08 13:16:42.976502 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:42.976485    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 13:16:42.986316 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:42.986297    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 13:16:42.997253 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:42.997225    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkd5f\" (UniqueName: \"kubernetes.io/projected/6fa64a88-f23c-42a6-8a1f-748e0278757a-kube-api-access-wkd5f\") pod \"network-check-target-br278\" (UID: \"6fa64a88-f23c-42a6-8a1f-748e0278757a\") " pod="openshift-network-diagnostics/network-check-target-br278"
Sep 08 13:16:43.158341 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:43.158300    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g5jt7"
Sep 08 13:16:43.171063 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:43.171046    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-x5pb6\""
Sep 08 13:16:43.179447 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:43.179424    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-br278"
Sep 08 13:16:43.434080 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:43.434010    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g5jt7"]
Sep 08 13:16:43.437177 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:16:43.437152    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a4cf01_77e6_44a0_8074_0929ee8dfee7.slice/crio-8fd2b32c7dbfe2765cf6bd982b3924bd44834790357be75fe5ef41771d71e539 WatchSource:0}: Error finding container 8fd2b32c7dbfe2765cf6bd982b3924bd44834790357be75fe5ef41771d71e539: Status 404 returned error can't find the container with id 8fd2b32c7dbfe2765cf6bd982b3924bd44834790357be75fe5ef41771d71e539
Sep 08 13:16:43.456107 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:43.456082    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-br278"]
Sep 08 13:16:43.459563 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:16:43.459541    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fa64a88_f23c_42a6_8a1f_748e0278757a.slice/crio-d05c31235535127e62cd5633a690b1af42ac9b2b5c260bcd686ab280d17850bc WatchSource:0}: Error finding container d05c31235535127e62cd5633a690b1af42ac9b2b5c260bcd686ab280d17850bc: Status 404 returned error can't find the container with id d05c31235535127e62cd5633a690b1af42ac9b2b5c260bcd686ab280d17850bc
Sep 08 13:16:44.298667 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:44.298628    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-br278" event={"ID":"6fa64a88-f23c-42a6-8a1f-748e0278757a","Type":"ContainerStarted","Data":"d05c31235535127e62cd5633a690b1af42ac9b2b5c260bcd686ab280d17850bc"}
Sep 08 13:16:44.300268 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:44.300235    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g5jt7" event={"ID":"24a4cf01-77e6-44a0-8074-0929ee8dfee7","Type":"ContainerStarted","Data":"8fd2b32c7dbfe2765cf6bd982b3924bd44834790357be75fe5ef41771d71e539"}
Sep 08 13:16:48.312565 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:48.312532    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-br278" event={"ID":"6fa64a88-f23c-42a6-8a1f-748e0278757a","Type":"ContainerStarted","Data":"d9053033ec12158390574b79bad4197c927ae23775800a6aa1d4641d24bfdc17"}
Sep 08 13:16:48.312933 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:48.312658    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-br278"
Sep 08 13:16:48.314056 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:48.314035    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g5jt7" event={"ID":"24a4cf01-77e6-44a0-8074-0929ee8dfee7","Type":"ContainerStarted","Data":"12d747e84d702c148784a5a53ef93a2771473b4d7493bdca324d05125d340af8"}
Sep 08 13:16:48.336217 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:48.335341    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-br278" podStartSLOduration=66.585542688 podStartE2EDuration="1m11.335321913s" totalImagesPullingTime="4.749779225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:15:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:48.33435824 +0000 UTC m=+71.786119172" watchObservedRunningTime="2026-09-08 13:16:48.335321913 +0000 UTC m=+71.787082845"
Sep 08 13:16:48.351492 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:16:48.351451    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-g5jt7" podStartSLOduration=65.575063942 podStartE2EDuration="1m10.351435222s" totalImagesPullingTime="4.77637128s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:48.350867925 +0000 UTC m=+71.802628855" watchObservedRunningTime="2026-09-08 13:16:48.351435222 +0000 UTC m=+71.803196164"
Sep 08 13:17:13.786149 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:17:13.786105    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bcc0442-53c5-4088-a419-10e4af66a483-metrics-tls\") pod \"dns-default-pvnnp\" (UID: \"7bcc0442-53c5-4088-a419-10e4af66a483\") " pod="openshift-dns/dns-default-pvnnp"
Sep 08 13:17:13.786149 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:17:13.786147    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-tls\") pod \"image-registry-bcb89cbc9-g9vr7\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") " pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:17:13.786815 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:17:13.786323    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:17:13.786815 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:17:13.786411    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bcc0442-53c5-4088-a419-10e4af66a483-metrics-tls podName:7bcc0442-53c5-4088-a419-10e4af66a483 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:17.786388144 +0000 UTC m=+161.238149077 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7bcc0442-53c5-4088-a419-10e4af66a483-metrics-tls") pod "dns-default-pvnnp" (UID: "7bcc0442-53c5-4088-a419-10e4af66a483") : secret "dns-default-metrics-tls" not found
Sep 08 13:17:13.786815 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:17:13.786332    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:17:13.786815 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:17:13.786434    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-bcb89cbc9-g9vr7: secret "image-registry-tls" not found
Sep 08 13:17:13.786815 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:17:13.786474    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-tls podName:494d9fb4-e6b1-4d95-a27d-194c4d4c6079 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:17.786461968 +0000 UTC m=+161.238222881 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-tls") pod "image-registry-bcb89cbc9-g9vr7" (UID: "494d9fb4-e6b1-4d95-a27d-194c4d4c6079") : secret "image-registry-tls" not found
Sep 08 13:17:19.319110 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:17:19.319077    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-br278"
Sep 08 13:17:46.807384 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:17:46.807329    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90dadd3f-c403-4d64-af3a-45b4c68163e5-metrics-certs\") pod \"network-metrics-daemon-q4m28\" (UID: \"90dadd3f-c403-4d64-af3a-45b4c68163e5\") " pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:17:46.807909 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:17:46.807493    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 08 13:17:46.807909 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:17:46.807576    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90dadd3f-c403-4d64-af3a-45b4c68163e5-metrics-certs podName:90dadd3f-c403-4d64-af3a-45b4c68163e5 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:48.807557799 +0000 UTC m=+252.259318713 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90dadd3f-c403-4d64-af3a-45b4c68163e5-metrics-certs") pod "network-metrics-daemon-q4m28" (UID: "90dadd3f-c403-4d64-af3a-45b4c68163e5") : secret "metrics-daemon-secret" not found
Sep 08 13:18:00.240499 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.240456    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kcvnc"]
Sep 08 13:18:00.243364 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.243346    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kcvnc"
Sep 08 13:18:00.246459 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.246431    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 08 13:18:00.247230 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.247209    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 08 13:18:00.247344 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.247280    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-hnkbm\""
Sep 08 13:18:00.248761 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.248743    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:18:00.256582 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.256556    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kcvnc"]
Sep 08 13:18:00.270338 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.270306    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-57dfddf5b-8t6lc"]
Sep 08 13:18:00.273262 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.273241    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57dfddf5b-8t6lc"
Sep 08 13:18:00.275648 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.275630    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 08 13:18:00.275648 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.275639    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 08 13:18:00.275818 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.275696    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 08 13:18:00.276107 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.276088    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 08 13:18:00.283620 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.283585    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-57dfddf5b-8t6lc"]
Sep 08 13:18:00.340401 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.340374    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-4q6fg"]
Sep 08 13:18:00.343317 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.343301    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-4q6fg"
Sep 08 13:18:00.345787 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.345765    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 08 13:18:00.345913 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.345792    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 08 13:18:00.345913 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.345892    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mspbv\""
Sep 08 13:18:00.346020 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.345972    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 08 13:18:00.346020 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.345985    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 08 13:18:00.351030 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.351013    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 08 13:18:00.353463 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.353436    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-4q6fg"]
Sep 08 13:18:00.402299 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.402267    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtpf5\" (UniqueName: \"kubernetes.io/projected/2ed17a76-47dc-4d02-9c80-18ce9f50860f-kube-api-access-qtpf5\") pod \"cluster-samples-operator-7cdcfcf76c-kcvnc\" (UID: \"2ed17a76-47dc-4d02-9c80-18ce9f50860f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kcvnc"
Sep 08 13:18:00.402299 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.402302    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/127c3838-69e5-45f5-b675-b6aa722d4a82-tmp\") pod \"klusterlet-addon-workmgr-57dfddf5b-8t6lc\" (UID: \"127c3838-69e5-45f5-b675-b6aa722d4a82\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57dfddf5b-8t6lc"
Sep 08 13:18:00.402471 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.402344    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj5f7\" (UniqueName: \"kubernetes.io/projected/127c3838-69e5-45f5-b675-b6aa722d4a82-kube-api-access-lj5f7\") pod \"klusterlet-addon-workmgr-57dfddf5b-8t6lc\" (UID: \"127c3838-69e5-45f5-b675-b6aa722d4a82\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57dfddf5b-8t6lc"
Sep 08 13:18:00.402471 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.402372    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ed17a76-47dc-4d02-9c80-18ce9f50860f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-kcvnc\" (UID: \"2ed17a76-47dc-4d02-9c80-18ce9f50860f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kcvnc"
Sep 08 13:18:00.402471 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.402401    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/127c3838-69e5-45f5-b675-b6aa722d4a82-klusterlet-config\") pod \"klusterlet-addon-workmgr-57dfddf5b-8t6lc\" (UID: \"127c3838-69e5-45f5-b675-b6aa722d4a82\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57dfddf5b-8t6lc"
Sep 08 13:18:00.503624 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.503525    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e818a776-d654-41ac-ba74-e47a0532f4d6-snapshots\") pod \"insights-operator-76bcd6ccf5-4q6fg\" (UID: \"e818a776-d654-41ac-ba74-e47a0532f4d6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4q6fg"
Sep 08 13:18:00.503624 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.503564    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ed17a76-47dc-4d02-9c80-18ce9f50860f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-kcvnc\" (UID: \"2ed17a76-47dc-4d02-9c80-18ce9f50860f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kcvnc"
Sep 08 13:18:00.503624 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.503581    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e818a776-d654-41ac-ba74-e47a0532f4d6-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-4q6fg\" (UID: \"e818a776-d654-41ac-ba74-e47a0532f4d6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4q6fg"
Sep 08 13:18:00.504071 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.503639    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/127c3838-69e5-45f5-b675-b6aa722d4a82-klusterlet-config\") pod \"klusterlet-addon-workmgr-57dfddf5b-8t6lc\" (UID: \"127c3838-69e5-45f5-b675-b6aa722d4a82\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57dfddf5b-8t6lc"
Sep 08 13:18:00.504071 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.503673    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e818a776-d654-41ac-ba74-e47a0532f4d6-tmp\") pod \"insights-operator-76bcd6ccf5-4q6fg\" (UID: \"e818a776-d654-41ac-ba74-e47a0532f4d6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4q6fg"
Sep 08 13:18:00.504071 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.503690    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e818a776-d654-41ac-ba74-e47a0532f4d6-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-4q6fg\" (UID: \"e818a776-d654-41ac-ba74-e47a0532f4d6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4q6fg"
Sep 08 13:18:00.504071 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:00.503705    2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 13:18:00.504071 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:00.503789    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ed17a76-47dc-4d02-9c80-18ce9f50860f-samples-operator-tls podName:2ed17a76-47dc-4d02-9c80-18ce9f50860f nodeName:}" failed. No retries permitted until 2026-09-08 13:18:01.003766233 +0000 UTC m=+144.455527173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2ed17a76-47dc-4d02-9c80-18ce9f50860f-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-kcvnc" (UID: "2ed17a76-47dc-4d02-9c80-18ce9f50860f") : secret "samples-operator-tls" not found
Sep 08 13:18:00.504071 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.503815    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e818a776-d654-41ac-ba74-e47a0532f4d6-serving-cert\") pod \"insights-operator-76bcd6ccf5-4q6fg\" (UID: \"e818a776-d654-41ac-ba74-e47a0532f4d6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4q6fg"
Sep 08 13:18:00.504071 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.503857    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgwdw\" (UniqueName: \"kubernetes.io/projected/e818a776-d654-41ac-ba74-e47a0532f4d6-kube-api-access-xgwdw\") pod \"insights-operator-76bcd6ccf5-4q6fg\" (UID: \"e818a776-d654-41ac-ba74-e47a0532f4d6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4q6fg"
Sep 08 13:18:00.504071 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.503889    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtpf5\" (UniqueName: \"kubernetes.io/projected/2ed17a76-47dc-4d02-9c80-18ce9f50860f-kube-api-access-qtpf5\") pod \"cluster-samples-operator-7cdcfcf76c-kcvnc\" (UID: \"2ed17a76-47dc-4d02-9c80-18ce9f50860f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kcvnc"
Sep 08 13:18:00.504071 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.503909    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/127c3838-69e5-45f5-b675-b6aa722d4a82-tmp\") pod \"klusterlet-addon-workmgr-57dfddf5b-8t6lc\" (UID: \"127c3838-69e5-45f5-b675-b6aa722d4a82\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57dfddf5b-8t6lc"
Sep 08 13:18:00.504071 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.503971    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj5f7\" (UniqueName: \"kubernetes.io/projected/127c3838-69e5-45f5-b675-b6aa722d4a82-kube-api-access-lj5f7\") pod \"klusterlet-addon-workmgr-57dfddf5b-8t6lc\" (UID: \"127c3838-69e5-45f5-b675-b6aa722d4a82\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57dfddf5b-8t6lc"
Sep 08 13:18:00.504437 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.504289    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/127c3838-69e5-45f5-b675-b6aa722d4a82-tmp\") pod \"klusterlet-addon-workmgr-57dfddf5b-8t6lc\" (UID: \"127c3838-69e5-45f5-b675-b6aa722d4a82\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57dfddf5b-8t6lc"
Sep 08 13:18:00.506152 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.506130    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/127c3838-69e5-45f5-b675-b6aa722d4a82-klusterlet-config\") pod \"klusterlet-addon-workmgr-57dfddf5b-8t6lc\" (UID: \"127c3838-69e5-45f5-b675-b6aa722d4a82\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57dfddf5b-8t6lc"
Sep 08 13:18:00.512326 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.512298    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtpf5\" (UniqueName: \"kubernetes.io/projected/2ed17a76-47dc-4d02-9c80-18ce9f50860f-kube-api-access-qtpf5\") pod \"cluster-samples-operator-7cdcfcf76c-kcvnc\" (UID: \"2ed17a76-47dc-4d02-9c80-18ce9f50860f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kcvnc"
Sep 08 13:18:00.512431 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.512349    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj5f7\" (UniqueName: \"kubernetes.io/projected/127c3838-69e5-45f5-b675-b6aa722d4a82-kube-api-access-lj5f7\") pod \"klusterlet-addon-workmgr-57dfddf5b-8t6lc\" (UID: \"127c3838-69e5-45f5-b675-b6aa722d4a82\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57dfddf5b-8t6lc"
Sep 08 13:18:00.582677 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.582640    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57dfddf5b-8t6lc"
Sep 08 13:18:00.604912 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.604885    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e818a776-d654-41ac-ba74-e47a0532f4d6-snapshots\") pod \"insights-operator-76bcd6ccf5-4q6fg\" (UID: \"e818a776-d654-41ac-ba74-e47a0532f4d6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4q6fg"
Sep 08 13:18:00.605021 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.604925    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e818a776-d654-41ac-ba74-e47a0532f4d6-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-4q6fg\" (UID: \"e818a776-d654-41ac-ba74-e47a0532f4d6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4q6fg"
Sep 08 13:18:00.605021 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.604956    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e818a776-d654-41ac-ba74-e47a0532f4d6-tmp\") pod \"insights-operator-76bcd6ccf5-4q6fg\" (UID: \"e818a776-d654-41ac-ba74-e47a0532f4d6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4q6fg"
Sep 08 13:18:00.605021 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.604972    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e818a776-d654-41ac-ba74-e47a0532f4d6-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-4q6fg\" (UID: \"e818a776-d654-41ac-ba74-e47a0532f4d6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4q6fg"
Sep 08 13:18:00.605021 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.604989    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e818a776-d654-41ac-ba74-e47a0532f4d6-serving-cert\") pod \"insights-operator-76bcd6ccf5-4q6fg\" (UID: \"e818a776-d654-41ac-ba74-e47a0532f4d6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4q6fg"
Sep 08 13:18:00.605021 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.605012    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgwdw\" (UniqueName: \"kubernetes.io/projected/e818a776-d654-41ac-ba74-e47a0532f4d6-kube-api-access-xgwdw\") pod \"insights-operator-76bcd6ccf5-4q6fg\" (UID: \"e818a776-d654-41ac-ba74-e47a0532f4d6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4q6fg"
Sep 08 13:18:00.605547 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.605520    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e818a776-d654-41ac-ba74-e47a0532f4d6-tmp\") pod \"insights-operator-76bcd6ccf5-4q6fg\" (UID: \"e818a776-d654-41ac-ba74-e47a0532f4d6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4q6fg"
Sep 08 13:18:00.605680 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.605544    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e818a776-d654-41ac-ba74-e47a0532f4d6-snapshots\") pod \"insights-operator-76bcd6ccf5-4q6fg\" (UID: \"e818a776-d654-41ac-ba74-e47a0532f4d6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4q6fg"
Sep 08 13:18:00.605680 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.605652    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e818a776-d654-41ac-ba74-e47a0532f4d6-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-4q6fg\" (UID: \"e818a776-d654-41ac-ba74-e47a0532f4d6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4q6fg"
Sep 08 13:18:00.605950 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.605926    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e818a776-d654-41ac-ba74-e47a0532f4d6-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-4q6fg\" (UID: \"e818a776-d654-41ac-ba74-e47a0532f4d6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4q6fg"
Sep 08 13:18:00.607339 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.607315    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e818a776-d654-41ac-ba74-e47a0532f4d6-serving-cert\") pod \"insights-operator-76bcd6ccf5-4q6fg\" (UID: \"e818a776-d654-41ac-ba74-e47a0532f4d6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4q6fg"
Sep 08 13:18:00.614098 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.614077    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgwdw\" (UniqueName: \"kubernetes.io/projected/e818a776-d654-41ac-ba74-e47a0532f4d6-kube-api-access-xgwdw\") pod \"insights-operator-76bcd6ccf5-4q6fg\" (UID: \"e818a776-d654-41ac-ba74-e47a0532f4d6\") " pod="openshift-insights/insights-operator-76bcd6ccf5-4q6fg"
Sep 08 13:18:00.653150 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.653119    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-4q6fg"
Sep 08 13:18:00.851701 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.851624    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-57dfddf5b-8t6lc"]
Sep 08 13:18:00.854345 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:18:00.854320    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod127c3838_69e5_45f5_b675_b6aa722d4a82.slice/crio-aaef46ab4cc443d891605d6f1fb34538a5ede9a5252a081ce6569dc856e2452d WatchSource:0}: Error finding container aaef46ab4cc443d891605d6f1fb34538a5ede9a5252a081ce6569dc856e2452d: Status 404 returned error can't find the container with id aaef46ab4cc443d891605d6f1fb34538a5ede9a5252a081ce6569dc856e2452d
Sep 08 13:18:00.919848 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:00.919819    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-4q6fg"]
Sep 08 13:18:00.922748 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:18:00.922725    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode818a776_d654_41ac_ba74_e47a0532f4d6.slice/crio-d40bf5371835a22fabaf456df1fb470cc570f075cc8328871537cbda5ec59558 WatchSource:0}: Error finding container d40bf5371835a22fabaf456df1fb470cc570f075cc8328871537cbda5ec59558: Status 404 returned error can't find the container with id d40bf5371835a22fabaf456df1fb470cc570f075cc8328871537cbda5ec59558
Sep 08 13:18:01.009199 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:01.009147    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ed17a76-47dc-4d02-9c80-18ce9f50860f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-kcvnc\" (UID: \"2ed17a76-47dc-4d02-9c80-18ce9f50860f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kcvnc"
Sep 08 13:18:01.009354 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:01.009271    2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 13:18:01.009354 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:01.009336    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ed17a76-47dc-4d02-9c80-18ce9f50860f-samples-operator-tls podName:2ed17a76-47dc-4d02-9c80-18ce9f50860f nodeName:}" failed. No retries permitted until 2026-09-08 13:18:02.009322358 +0000 UTC m=+145.461083267 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2ed17a76-47dc-4d02-9c80-18ce9f50860f-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-kcvnc" (UID: "2ed17a76-47dc-4d02-9c80-18ce9f50860f") : secret "samples-operator-tls" not found
Sep 08 13:18:01.482851 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:01.482805    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57dfddf5b-8t6lc" event={"ID":"127c3838-69e5-45f5-b675-b6aa722d4a82","Type":"ContainerStarted","Data":"aaef46ab4cc443d891605d6f1fb34538a5ede9a5252a081ce6569dc856e2452d"}
Sep 08 13:18:01.483873 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:01.483852    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-4q6fg" event={"ID":"e818a776-d654-41ac-ba74-e47a0532f4d6","Type":"ContainerStarted","Data":"d40bf5371835a22fabaf456df1fb470cc570f075cc8328871537cbda5ec59558"}
Sep 08 13:18:02.014527 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:02.014492    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ed17a76-47dc-4d02-9c80-18ce9f50860f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-kcvnc\" (UID: \"2ed17a76-47dc-4d02-9c80-18ce9f50860f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kcvnc"
Sep 08 13:18:02.014722 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:02.014637    2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 13:18:02.014722 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:02.014693    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ed17a76-47dc-4d02-9c80-18ce9f50860f-samples-operator-tls podName:2ed17a76-47dc-4d02-9c80-18ce9f50860f nodeName:}" failed. No retries permitted until 2026-09-08 13:18:04.014679112 +0000 UTC m=+147.466440020 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2ed17a76-47dc-4d02-9c80-18ce9f50860f-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-kcvnc" (UID: "2ed17a76-47dc-4d02-9c80-18ce9f50860f") : secret "samples-operator-tls" not found
Sep 08 13:18:04.029932 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:04.029884    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ed17a76-47dc-4d02-9c80-18ce9f50860f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-kcvnc\" (UID: \"2ed17a76-47dc-4d02-9c80-18ce9f50860f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kcvnc"
Sep 08 13:18:04.030411 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:04.030110    2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 13:18:04.030411 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:04.030196    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ed17a76-47dc-4d02-9c80-18ce9f50860f-samples-operator-tls podName:2ed17a76-47dc-4d02-9c80-18ce9f50860f nodeName:}" failed. No retries permitted until 2026-09-08 13:18:08.030173681 +0000 UTC m=+151.481934593 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2ed17a76-47dc-4d02-9c80-18ce9f50860f-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-kcvnc" (UID: "2ed17a76-47dc-4d02-9c80-18ce9f50860f") : secret "samples-operator-tls" not found
Sep 08 13:18:04.494479 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:04.494407    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-4q6fg" event={"ID":"e818a776-d654-41ac-ba74-e47a0532f4d6","Type":"ContainerStarted","Data":"0c66008a3b78ff5c44a7a94f49775aa94580d16f2a8783f99f2b4b11935f2ff0"}
Sep 08 13:18:04.515153 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:04.515091    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-4q6fg" podStartSLOduration=1.8986892549999999 podStartE2EDuration="4.515068523s" totalImagesPullingTime="2.616379268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:18:04.513271707 +0000 UTC m=+147.965032637" watchObservedRunningTime="2026-09-08 13:18:04.515068523 +0000 UTC m=+147.966829455"
Sep 08 13:18:05.537501 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:05.537464    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8nbzm"]
Sep 08 13:18:05.540975 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:05.540948    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8nbzm"
Sep 08 13:18:05.543535 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:05.543340    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 08 13:18:05.543535 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:05.543413    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-x5h6q\""
Sep 08 13:18:05.543751 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:05.543702    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 08 13:18:05.564810 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:05.564784    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8nbzm"]
Sep 08 13:18:05.644225 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:05.644190    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5872b485-7dfa-45e1-8313-2393fc7361e6-crio-socket\") pod \"insights-runtime-extractor-8nbzm\" (UID: \"5872b485-7dfa-45e1-8313-2393fc7361e6\") " pod="openshift-insights/insights-runtime-extractor-8nbzm"
Sep 08 13:18:05.644225 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:05.644229    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5872b485-7dfa-45e1-8313-2393fc7361e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8nbzm\" (UID: \"5872b485-7dfa-45e1-8313-2393fc7361e6\") " pod="openshift-insights/insights-runtime-extractor-8nbzm"
Sep 08 13:18:05.644493 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:05.644300    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5872b485-7dfa-45e1-8313-2393fc7361e6-data-volume\") pod \"insights-runtime-extractor-8nbzm\" (UID: \"5872b485-7dfa-45e1-8313-2393fc7361e6\") " pod="openshift-insights/insights-runtime-extractor-8nbzm"
Sep 08 13:18:05.644534 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:05.644505    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6s9r\" (UniqueName: \"kubernetes.io/projected/5872b485-7dfa-45e1-8313-2393fc7361e6-kube-api-access-q6s9r\") pod \"insights-runtime-extractor-8nbzm\" (UID: \"5872b485-7dfa-45e1-8313-2393fc7361e6\") " pod="openshift-insights/insights-runtime-extractor-8nbzm"
Sep 08 13:18:05.644581 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:05.644542    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5872b485-7dfa-45e1-8313-2393fc7361e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8nbzm\" (UID: \"5872b485-7dfa-45e1-8313-2393fc7361e6\") " pod="openshift-insights/insights-runtime-extractor-8nbzm"
Sep 08 13:18:05.745421 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:05.745380    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6s9r\" (UniqueName: \"kubernetes.io/projected/5872b485-7dfa-45e1-8313-2393fc7361e6-kube-api-access-q6s9r\") pod \"insights-runtime-extractor-8nbzm\" (UID: \"5872b485-7dfa-45e1-8313-2393fc7361e6\") " pod="openshift-insights/insights-runtime-extractor-8nbzm"
Sep 08 13:18:05.745421 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:05.745423    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5872b485-7dfa-45e1-8313-2393fc7361e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8nbzm\" (UID: \"5872b485-7dfa-45e1-8313-2393fc7361e6\") " pod="openshift-insights/insights-runtime-extractor-8nbzm"
Sep 08 13:18:05.745708 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:05.745461    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5872b485-7dfa-45e1-8313-2393fc7361e6-crio-socket\") pod \"insights-runtime-extractor-8nbzm\" (UID: \"5872b485-7dfa-45e1-8313-2393fc7361e6\") " pod="openshift-insights/insights-runtime-extractor-8nbzm"
Sep 08 13:18:05.745708 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:05.745484    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5872b485-7dfa-45e1-8313-2393fc7361e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8nbzm\" (UID: \"5872b485-7dfa-45e1-8313-2393fc7361e6\") " pod="openshift-insights/insights-runtime-extractor-8nbzm"
Sep 08 13:18:05.745708 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:05.745510    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5872b485-7dfa-45e1-8313-2393fc7361e6-data-volume\") pod \"insights-runtime-extractor-8nbzm\" (UID: \"5872b485-7dfa-45e1-8313-2393fc7361e6\") " pod="openshift-insights/insights-runtime-extractor-8nbzm"
Sep 08 13:18:05.745708 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:05.745553    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:18:05.745708 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:05.745564    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5872b485-7dfa-45e1-8313-2393fc7361e6-crio-socket\") pod \"insights-runtime-extractor-8nbzm\" (UID: \"5872b485-7dfa-45e1-8313-2393fc7361e6\") " pod="openshift-insights/insights-runtime-extractor-8nbzm"
Sep 08 13:18:05.745708 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:05.745622    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:05.745708 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:05.745680    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5872b485-7dfa-45e1-8313-2393fc7361e6-kube-rbac-proxy-cm podName:5872b485-7dfa-45e1-8313-2393fc7361e6 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:06.24565831 +0000 UTC m=+149.697419237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5872b485-7dfa-45e1-8313-2393fc7361e6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8nbzm" (UID: "5872b485-7dfa-45e1-8313-2393fc7361e6") : configmap "kube-rbac-proxy" not found
Sep 08 13:18:05.745708 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:05.745703    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5872b485-7dfa-45e1-8313-2393fc7361e6-insights-runtime-extractor-tls podName:5872b485-7dfa-45e1-8313-2393fc7361e6 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:06.24569251 +0000 UTC m=+149.697453418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5872b485-7dfa-45e1-8313-2393fc7361e6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8nbzm" (UID: "5872b485-7dfa-45e1-8313-2393fc7361e6") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:05.746103 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:05.745980    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5872b485-7dfa-45e1-8313-2393fc7361e6-data-volume\") pod \"insights-runtime-extractor-8nbzm\" (UID: \"5872b485-7dfa-45e1-8313-2393fc7361e6\") " pod="openshift-insights/insights-runtime-extractor-8nbzm"
Sep 08 13:18:05.754584 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:05.754554    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6s9r\" (UniqueName: \"kubernetes.io/projected/5872b485-7dfa-45e1-8313-2393fc7361e6-kube-api-access-q6s9r\") pod \"insights-runtime-extractor-8nbzm\" (UID: \"5872b485-7dfa-45e1-8313-2393fc7361e6\") " pod="openshift-insights/insights-runtime-extractor-8nbzm"
Sep 08 13:18:06.249740 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:06.249700    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5872b485-7dfa-45e1-8313-2393fc7361e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8nbzm\" (UID: \"5872b485-7dfa-45e1-8313-2393fc7361e6\") " pod="openshift-insights/insights-runtime-extractor-8nbzm"
Sep 08 13:18:06.249740 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:06.249750    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5872b485-7dfa-45e1-8313-2393fc7361e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8nbzm\" (UID: \"5872b485-7dfa-45e1-8313-2393fc7361e6\") " pod="openshift-insights/insights-runtime-extractor-8nbzm"
Sep 08 13:18:06.250002 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:06.249829    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:18:06.250002 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:06.249867    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:06.250002 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:06.249921    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5872b485-7dfa-45e1-8313-2393fc7361e6-kube-rbac-proxy-cm podName:5872b485-7dfa-45e1-8313-2393fc7361e6 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:07.24990287 +0000 UTC m=+150.701663779 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5872b485-7dfa-45e1-8313-2393fc7361e6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8nbzm" (UID: "5872b485-7dfa-45e1-8313-2393fc7361e6") : configmap "kube-rbac-proxy" not found
Sep 08 13:18:06.250002 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:06.249936    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5872b485-7dfa-45e1-8313-2393fc7361e6-insights-runtime-extractor-tls podName:5872b485-7dfa-45e1-8313-2393fc7361e6 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:07.249930158 +0000 UTC m=+150.701691067 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5872b485-7dfa-45e1-8313-2393fc7361e6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8nbzm" (UID: "5872b485-7dfa-45e1-8313-2393fc7361e6") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:06.630349 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:06.630319    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mzj2b"]
Sep 08 13:18:06.633169 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:06.633152    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mzj2b"
Sep 08 13:18:06.635554 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:06.635533    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 08 13:18:06.635885 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:06.635866    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 08 13:18:06.636513 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:06.636501    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ksdrh\""
Sep 08 13:18:06.642919 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:06.642899    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mzj2b"]
Sep 08 13:18:06.754653 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:06.754579    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgh8b\" (UniqueName: \"kubernetes.io/projected/d5835194-1443-41e5-931f-4bc84fc9c9c7-kube-api-access-kgh8b\") pod \"migrator-fdbb8fdff-mzj2b\" (UID: \"d5835194-1443-41e5-931f-4bc84fc9c9c7\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mzj2b"
Sep 08 13:18:06.855985 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:06.855947    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgh8b\" (UniqueName: \"kubernetes.io/projected/d5835194-1443-41e5-931f-4bc84fc9c9c7-kube-api-access-kgh8b\") pod \"migrator-fdbb8fdff-mzj2b\" (UID: \"d5835194-1443-41e5-931f-4bc84fc9c9c7\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mzj2b"
Sep 08 13:18:06.865699 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:06.865667    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgh8b\" (UniqueName: \"kubernetes.io/projected/d5835194-1443-41e5-931f-4bc84fc9c9c7-kube-api-access-kgh8b\") pod \"migrator-fdbb8fdff-mzj2b\" (UID: \"d5835194-1443-41e5-931f-4bc84fc9c9c7\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mzj2b"
Sep 08 13:18:06.941914 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:06.941828    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mzj2b"
Sep 08 13:18:07.209494 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:07.209399    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mzj2b"]
Sep 08 13:18:07.213225 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:18:07.213188    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5835194_1443_41e5_931f_4bc84fc9c9c7.slice/crio-eec51aa32a771ea99ab43bef9005de6ee2bcc9a977b44c5dc968d615968ce211 WatchSource:0}: Error finding container eec51aa32a771ea99ab43bef9005de6ee2bcc9a977b44c5dc968d615968ce211: Status 404 returned error can't find the container with id eec51aa32a771ea99ab43bef9005de6ee2bcc9a977b44c5dc968d615968ce211
Sep 08 13:18:07.259559 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:07.259531    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5872b485-7dfa-45e1-8313-2393fc7361e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8nbzm\" (UID: \"5872b485-7dfa-45e1-8313-2393fc7361e6\") " pod="openshift-insights/insights-runtime-extractor-8nbzm"
Sep 08 13:18:07.259751 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:07.259575    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5872b485-7dfa-45e1-8313-2393fc7361e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8nbzm\" (UID: \"5872b485-7dfa-45e1-8313-2393fc7361e6\") " pod="openshift-insights/insights-runtime-extractor-8nbzm"
Sep 08 13:18:07.259751 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:07.259697    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:07.259751 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:07.259720    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:18:07.259751 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:07.259745    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5872b485-7dfa-45e1-8313-2393fc7361e6-insights-runtime-extractor-tls podName:5872b485-7dfa-45e1-8313-2393fc7361e6 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:09.259731648 +0000 UTC m=+152.711492556 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5872b485-7dfa-45e1-8313-2393fc7361e6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8nbzm" (UID: "5872b485-7dfa-45e1-8313-2393fc7361e6") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:07.259917 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:07.259767    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5872b485-7dfa-45e1-8313-2393fc7361e6-kube-rbac-proxy-cm podName:5872b485-7dfa-45e1-8313-2393fc7361e6 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:09.25975522 +0000 UTC m=+152.711516129 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5872b485-7dfa-45e1-8313-2393fc7361e6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8nbzm" (UID: "5872b485-7dfa-45e1-8313-2393fc7361e6") : configmap "kube-rbac-proxy" not found
Sep 08 13:18:07.507292 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:07.507195    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57dfddf5b-8t6lc" event={"ID":"127c3838-69e5-45f5-b675-b6aa722d4a82","Type":"ContainerStarted","Data":"9e32c77f81b58acdfb7de5801e20d24be5e23d636add75ebc79e82fbfdc6ed04"}
Sep 08 13:18:07.507458 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:07.507406    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57dfddf5b-8t6lc"
Sep 08 13:18:07.508497 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:07.508466    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mzj2b" event={"ID":"d5835194-1443-41e5-931f-4bc84fc9c9c7","Type":"ContainerStarted","Data":"eec51aa32a771ea99ab43bef9005de6ee2bcc9a977b44c5dc968d615968ce211"}
Sep 08 13:18:07.509100 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:07.509082    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57dfddf5b-8t6lc"
Sep 08 13:18:07.524581 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:07.524532    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-57dfddf5b-8t6lc" podStartSLOduration=1.707116784 podStartE2EDuration="7.524516801s" totalImagesPullingTime="5.817400017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:18:07.523844347 +0000 UTC m=+150.975605280" watchObservedRunningTime="2026-09-08 13:18:07.524516801 +0000 UTC m=+150.976277732"
Sep 08 13:18:07.692184 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:07.692159    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vwvwg_3ce2ec68-0166-484d-8e1a-28cc99334e5d/dns-node-resolver/0.log"
Sep 08 13:18:08.067745 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:08.067702    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ed17a76-47dc-4d02-9c80-18ce9f50860f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-kcvnc\" (UID: \"2ed17a76-47dc-4d02-9c80-18ce9f50860f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kcvnc"
Sep 08 13:18:08.067929 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:08.067864    2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 08 13:18:08.067982 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:08.067945    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ed17a76-47dc-4d02-9c80-18ce9f50860f-samples-operator-tls podName:2ed17a76-47dc-4d02-9c80-18ce9f50860f nodeName:}" failed. No retries permitted until 2026-09-08 13:18:16.067924432 +0000 UTC m=+159.519685355 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2ed17a76-47dc-4d02-9c80-18ce9f50860f-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-kcvnc" (UID: "2ed17a76-47dc-4d02-9c80-18ce9f50860f") : secret "samples-operator-tls" not found
Sep 08 13:18:08.492403 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:08.492382    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7t44n_eb60fb08-544c-4bd8-93fe-103f5f2bde69/node-ca/0.log"
Sep 08 13:18:09.092800 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:09.092766    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-d2mvr_349aa96b-eaec-45cf-9a20-7f2227273c2c/serve-healthcheck-canary/0.log"
Sep 08 13:18:09.277547 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:09.277509    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5872b485-7dfa-45e1-8313-2393fc7361e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8nbzm\" (UID: \"5872b485-7dfa-45e1-8313-2393fc7361e6\") " pod="openshift-insights/insights-runtime-extractor-8nbzm"
Sep 08 13:18:09.277725 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:09.277615    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5872b485-7dfa-45e1-8313-2393fc7361e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8nbzm\" (UID: \"5872b485-7dfa-45e1-8313-2393fc7361e6\") " pod="openshift-insights/insights-runtime-extractor-8nbzm"
Sep 08 13:18:09.277725 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:09.277689    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:18:09.277725 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:09.277689    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:09.277828 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:09.277741    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5872b485-7dfa-45e1-8313-2393fc7361e6-kube-rbac-proxy-cm podName:5872b485-7dfa-45e1-8313-2393fc7361e6 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:13.277727406 +0000 UTC m=+156.729488315 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5872b485-7dfa-45e1-8313-2393fc7361e6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8nbzm" (UID: "5872b485-7dfa-45e1-8313-2393fc7361e6") : configmap "kube-rbac-proxy" not found
Sep 08 13:18:09.277828 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:09.277754    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5872b485-7dfa-45e1-8313-2393fc7361e6-insights-runtime-extractor-tls podName:5872b485-7dfa-45e1-8313-2393fc7361e6 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:13.277747876 +0000 UTC m=+156.729508785 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5872b485-7dfa-45e1-8313-2393fc7361e6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8nbzm" (UID: "5872b485-7dfa-45e1-8313-2393fc7361e6") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:09.516088 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:09.516050    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mzj2b" event={"ID":"d5835194-1443-41e5-931f-4bc84fc9c9c7","Type":"ContainerStarted","Data":"a2ad05b7e16633855a1080c4c1b2d9abdcd69f34c498d34db73d8428f3994d94"}
Sep 08 13:18:09.516282 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:09.516112    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mzj2b" event={"ID":"d5835194-1443-41e5-931f-4bc84fc9c9c7","Type":"ContainerStarted","Data":"c225f30282e39433c97a7599f4874c2eb6a3f18f600926a9f1bf0a07b35fee6a"}
Sep 08 13:18:09.531674 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:09.531583    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-mzj2b" podStartSLOduration=2.276858016 podStartE2EDuration="3.531568065s" totalImagesPullingTime="1.254710049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:18:09.530506587 +0000 UTC m=+152.982267518" watchObservedRunningTime="2026-09-08 13:18:09.531568065 +0000 UTC m=+152.983328996"
Sep 08 13:18:12.938097 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:12.938054    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7" podUID="494d9fb4-e6b1-4d95-a27d-194c4d4c6079"
Sep 08 13:18:12.967432 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:12.967383    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-pvnnp" podUID="7bcc0442-53c5-4088-a419-10e4af66a483"
Sep 08 13:18:13.310705 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:13.310611    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5872b485-7dfa-45e1-8313-2393fc7361e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8nbzm\" (UID: \"5872b485-7dfa-45e1-8313-2393fc7361e6\") " pod="openshift-insights/insights-runtime-extractor-8nbzm"
Sep 08 13:18:13.310705 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:13.310663    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5872b485-7dfa-45e1-8313-2393fc7361e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8nbzm\" (UID: \"5872b485-7dfa-45e1-8313-2393fc7361e6\") " pod="openshift-insights/insights-runtime-extractor-8nbzm"
Sep 08 13:18:13.310901 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:13.310755    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:18:13.310901 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:13.310805    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:13.310901 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:13.310835    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5872b485-7dfa-45e1-8313-2393fc7361e6-kube-rbac-proxy-cm podName:5872b485-7dfa-45e1-8313-2393fc7361e6 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:21.310816612 +0000 UTC m=+164.762577537 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5872b485-7dfa-45e1-8313-2393fc7361e6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8nbzm" (UID: "5872b485-7dfa-45e1-8313-2393fc7361e6") : configmap "kube-rbac-proxy" not found
Sep 08 13:18:13.310901 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:13.310850    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5872b485-7dfa-45e1-8313-2393fc7361e6-insights-runtime-extractor-tls podName:5872b485-7dfa-45e1-8313-2393fc7361e6 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:21.310843846 +0000 UTC m=+164.762604756 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5872b485-7dfa-45e1-8313-2393fc7361e6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8nbzm" (UID: "5872b485-7dfa-45e1-8313-2393fc7361e6") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:13.529038 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:13.529011    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pvnnp"
Sep 08 13:18:13.529179 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:13.529011    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:18:14.063433 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:14.063391    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-q4m28" podUID="90dadd3f-c403-4d64-af3a-45b4c68163e5"
Sep 08 13:18:16.132763 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:16.132719    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ed17a76-47dc-4d02-9c80-18ce9f50860f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-kcvnc\" (UID: \"2ed17a76-47dc-4d02-9c80-18ce9f50860f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kcvnc"
Sep 08 13:18:16.135146 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:16.135114    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ed17a76-47dc-4d02-9c80-18ce9f50860f-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-kcvnc\" (UID: \"2ed17a76-47dc-4d02-9c80-18ce9f50860f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kcvnc"
Sep 08 13:18:16.152802 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:16.152772    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kcvnc"
Sep 08 13:18:16.425200 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:16.425119    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kcvnc"]
Sep 08 13:18:16.538972 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:16.538933    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kcvnc" event={"ID":"2ed17a76-47dc-4d02-9c80-18ce9f50860f","Type":"ContainerStarted","Data":"83fc8e13d577cdf32b8b44f3d2365ed2a3e3c96f6dfd702dd011adf54209b49f"}
Sep 08 13:18:17.846854 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:17.846801    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-tls\") pod \"image-registry-bcb89cbc9-g9vr7\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") " pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:18:17.847367 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:17.846895    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bcc0442-53c5-4088-a419-10e4af66a483-metrics-tls\") pod \"dns-default-pvnnp\" (UID: \"7bcc0442-53c5-4088-a419-10e4af66a483\") " pod="openshift-dns/dns-default-pvnnp"
Sep 08 13:18:17.849826 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:17.849802    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7bcc0442-53c5-4088-a419-10e4af66a483-metrics-tls\") pod \"dns-default-pvnnp\" (UID: \"7bcc0442-53c5-4088-a419-10e4af66a483\") " pod="openshift-dns/dns-default-pvnnp"
Sep 08 13:18:17.850093 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:17.850070    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-tls\") pod \"image-registry-bcb89cbc9-g9vr7\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") " pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:18:18.032957 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:18.032918    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wqhr2\""
Sep 08 13:18:18.032957 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:18.032920    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8shlb\""
Sep 08 13:18:18.040771 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:18.040738    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:18:18.040924 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:18.040784    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pvnnp"
Sep 08 13:18:18.402118 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:18.402042    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pvnnp"]
Sep 08 13:18:18.404860 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:18:18.404835    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bcc0442_53c5_4088_a419_10e4af66a483.slice/crio-238b8922924418ee4764db888019cac99ab2533252793a3aa89cc8dd105d71a7 WatchSource:0}: Error finding container 238b8922924418ee4764db888019cac99ab2533252793a3aa89cc8dd105d71a7: Status 404 returned error can't find the container with id 238b8922924418ee4764db888019cac99ab2533252793a3aa89cc8dd105d71a7
Sep 08 13:18:18.409086 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:18.409064    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-bcb89cbc9-g9vr7"]
Sep 08 13:18:18.412279 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:18:18.412257    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod494d9fb4_e6b1_4d95_a27d_194c4d4c6079.slice/crio-012d1e400e3508b88e02cbe424f7c391bf56aba1a9a1fdd60f36ad4a80ea08e7 WatchSource:0}: Error finding container 012d1e400e3508b88e02cbe424f7c391bf56aba1a9a1fdd60f36ad4a80ea08e7: Status 404 returned error can't find the container with id 012d1e400e3508b88e02cbe424f7c391bf56aba1a9a1fdd60f36ad4a80ea08e7
Sep 08 13:18:18.547583 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:18.547548    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kcvnc" event={"ID":"2ed17a76-47dc-4d02-9c80-18ce9f50860f","Type":"ContainerStarted","Data":"a8a19190439391f28a17df6f8493b7aee84bd773042b65d7263b377056c4d087"}
Sep 08 13:18:18.547762 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:18.547633    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kcvnc" event={"ID":"2ed17a76-47dc-4d02-9c80-18ce9f50860f","Type":"ContainerStarted","Data":"ba694e5430093e50d0eab2f7ebd6a737bb249b9f2cd0d71fce992f357e170e00"}
Sep 08 13:18:18.549173 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:18.549142    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7" event={"ID":"494d9fb4-e6b1-4d95-a27d-194c4d4c6079","Type":"ContainerStarted","Data":"7be584a87cefbed42a75e29ce4e097d62b9447199a7b8d32d0887a67c19920b9"}
Sep 08 13:18:18.549279 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:18.549187    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7" event={"ID":"494d9fb4-e6b1-4d95-a27d-194c4d4c6079","Type":"ContainerStarted","Data":"012d1e400e3508b88e02cbe424f7c391bf56aba1a9a1fdd60f36ad4a80ea08e7"}
Sep 08 13:18:18.549279 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:18.549237    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:18:18.550389 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:18.550364    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pvnnp" event={"ID":"7bcc0442-53c5-4088-a419-10e4af66a483","Type":"ContainerStarted","Data":"238b8922924418ee4764db888019cac99ab2533252793a3aa89cc8dd105d71a7"}
Sep 08 13:18:18.565829 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:18.565783    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kcvnc" podStartSLOduration=16.847225879 podStartE2EDuration="18.565769165s" totalImagesPullingTime="1.718543286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:18:18.564790642 +0000 UTC m=+162.016551575" watchObservedRunningTime="2026-09-08 13:18:18.565769165 +0000 UTC m=+162.017530095"
Sep 08 13:18:18.587670 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:18.587568    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7" podStartSLOduration=160.58754504 podStartE2EDuration="2m40.58754504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:18:18.58653742 +0000 UTC m=+162.038298380" watchObservedRunningTime="2026-09-08 13:18:18.58754504 +0000 UTC m=+162.039305973"
Sep 08 13:18:21.375803 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:21.375756    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5872b485-7dfa-45e1-8313-2393fc7361e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8nbzm\" (UID: \"5872b485-7dfa-45e1-8313-2393fc7361e6\") " pod="openshift-insights/insights-runtime-extractor-8nbzm"
Sep 08 13:18:21.376179 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:21.375818    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5872b485-7dfa-45e1-8313-2393fc7361e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8nbzm\" (UID: \"5872b485-7dfa-45e1-8313-2393fc7361e6\") " pod="openshift-insights/insights-runtime-extractor-8nbzm"
Sep 08 13:18:21.376179 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:21.375958    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:21.376179 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:21.376028    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5872b485-7dfa-45e1-8313-2393fc7361e6-insights-runtime-extractor-tls podName:5872b485-7dfa-45e1-8313-2393fc7361e6 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:37.376008734 +0000 UTC m=+180.827769648 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5872b485-7dfa-45e1-8313-2393fc7361e6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8nbzm" (UID: "5872b485-7dfa-45e1-8313-2393fc7361e6") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:21.376456 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:21.376435    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5872b485-7dfa-45e1-8313-2393fc7361e6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8nbzm\" (UID: \"5872b485-7dfa-45e1-8313-2393fc7361e6\") " pod="openshift-insights/insights-runtime-extractor-8nbzm"
Sep 08 13:18:21.562388 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:21.562351    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pvnnp" event={"ID":"7bcc0442-53c5-4088-a419-10e4af66a483","Type":"ContainerStarted","Data":"16d1a8358abd8706847aac4b0ef6ab674c1822f4368660990f459cf411501de0"}
Sep 08 13:18:21.562549 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:21.562414    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pvnnp" event={"ID":"7bcc0442-53c5-4088-a419-10e4af66a483","Type":"ContainerStarted","Data":"f16bec9f5001bb0873d5f9471f2129edd4ed42fb933c87c3148aa8a587bd48b9"}
Sep 08 13:18:21.562549 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:21.562442    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pvnnp"
Sep 08 13:18:21.580885 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:21.580836    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pvnnp" podStartSLOduration=130.493663001 podStartE2EDuration="2m12.58081394s" totalImagesPullingTime="2.087150939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:18:21.579760229 +0000 UTC m=+165.031521161" watchObservedRunningTime="2026-09-08 13:18:21.58081394 +0000 UTC m=+165.032574862"
Sep 08 13:18:25.048187 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:25.048141    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:18:29.199320 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:29.199244    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-hz955"]
Sep 08 13:18:29.202189 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:29.202173    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-hz955"
Sep 08 13:18:29.204441 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:29.204419    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 08 13:18:29.204576 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:29.204470    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 08 13:18:29.204576 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:29.204513    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-2f69t\""
Sep 08 13:18:29.214950 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:29.214928    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-hz955"]
Sep 08 13:18:29.229239 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:29.229214    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqnx8\" (UniqueName: \"kubernetes.io/projected/46b98f99-3985-4376-818e-269810605864-kube-api-access-zqnx8\") pod \"downloads-85d98c998f-hz955\" (UID: \"46b98f99-3985-4376-818e-269810605864\") " pod="openshift-console/downloads-85d98c998f-hz955"
Sep 08 13:18:29.329893 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:29.329857    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqnx8\" (UniqueName: \"kubernetes.io/projected/46b98f99-3985-4376-818e-269810605864-kube-api-access-zqnx8\") pod \"downloads-85d98c998f-hz955\" (UID: \"46b98f99-3985-4376-818e-269810605864\") " pod="openshift-console/downloads-85d98c998f-hz955"
Sep 08 13:18:29.337942 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:29.337909    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqnx8\" (UniqueName: \"kubernetes.io/projected/46b98f99-3985-4376-818e-269810605864-kube-api-access-zqnx8\") pod \"downloads-85d98c998f-hz955\" (UID: \"46b98f99-3985-4376-818e-269810605864\") " pod="openshift-console/downloads-85d98c998f-hz955"
Sep 08 13:18:29.511420 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:29.511335    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-hz955"
Sep 08 13:18:29.797986 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:29.797887    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-hz955"]
Sep 08 13:18:29.801858 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:18:29.801831    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b98f99_3985_4376_818e_269810605864.slice/crio-62a8296edfaf8cc4266259e76d8b14989c04208227b95b006094e521fff61d83 WatchSource:0}: Error finding container 62a8296edfaf8cc4266259e76d8b14989c04208227b95b006094e521fff61d83: Status 404 returned error can't find the container with id 62a8296edfaf8cc4266259e76d8b14989c04208227b95b006094e521fff61d83
Sep 08 13:18:30.596160 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:30.596120    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-hz955" event={"ID":"46b98f99-3985-4376-818e-269810605864","Type":"ContainerStarted","Data":"62a8296edfaf8cc4266259e76d8b14989c04208227b95b006094e521fff61d83"}
Sep 08 13:18:31.569462 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:31.569429    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pvnnp"
Sep 08 13:18:36.961338 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:36.961306    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-576bd68b6b-qpsch"]
Sep 08 13:18:36.964662 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:36.964631    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-576bd68b6b-qpsch"
Sep 08 13:18:36.967267 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:36.967242    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 08 13:18:36.967267 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:36.967267    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kzhlx\""
Sep 08 13:18:36.968086 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:36.968038    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 08 13:18:36.968289 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:36.968086    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 08 13:18:36.968289 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:36.968135    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 08 13:18:36.968289 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:36.968038    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 08 13:18:36.974712 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:36.974690    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-576bd68b6b-qpsch"]
Sep 08 13:18:36.996787 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:36.996612    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eac540e7-77f1-49ba-b3da-b875082bcaea-oauth-serving-cert\") pod \"console-576bd68b6b-qpsch\" (UID: \"eac540e7-77f1-49ba-b3da-b875082bcaea\") " pod="openshift-console/console-576bd68b6b-qpsch"
Sep 08 13:18:36.996787 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:36.996664    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eac540e7-77f1-49ba-b3da-b875082bcaea-console-oauth-config\") pod \"console-576bd68b6b-qpsch\" (UID: \"eac540e7-77f1-49ba-b3da-b875082bcaea\") " pod="openshift-console/console-576bd68b6b-qpsch"
Sep 08 13:18:36.996787 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:36.996702    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eac540e7-77f1-49ba-b3da-b875082bcaea-console-serving-cert\") pod \"console-576bd68b6b-qpsch\" (UID: \"eac540e7-77f1-49ba-b3da-b875082bcaea\") " pod="openshift-console/console-576bd68b6b-qpsch"
Sep 08 13:18:36.996787 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:36.996720    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eac540e7-77f1-49ba-b3da-b875082bcaea-service-ca\") pod \"console-576bd68b6b-qpsch\" (UID: \"eac540e7-77f1-49ba-b3da-b875082bcaea\") " pod="openshift-console/console-576bd68b6b-qpsch"
Sep 08 13:18:36.997045 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:36.996815    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eac540e7-77f1-49ba-b3da-b875082bcaea-console-config\") pod \"console-576bd68b6b-qpsch\" (UID: \"eac540e7-77f1-49ba-b3da-b875082bcaea\") " pod="openshift-console/console-576bd68b6b-qpsch"
Sep 08 13:18:36.997045 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:36.996838    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwcqk\" (UniqueName: \"kubernetes.io/projected/eac540e7-77f1-49ba-b3da-b875082bcaea-kube-api-access-fwcqk\") pod \"console-576bd68b6b-qpsch\" (UID: \"eac540e7-77f1-49ba-b3da-b875082bcaea\") " pod="openshift-console/console-576bd68b6b-qpsch"
Sep 08 13:18:37.097426 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:37.097393    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eac540e7-77f1-49ba-b3da-b875082bcaea-console-config\") pod \"console-576bd68b6b-qpsch\" (UID: \"eac540e7-77f1-49ba-b3da-b875082bcaea\") " pod="openshift-console/console-576bd68b6b-qpsch"
Sep 08 13:18:37.097651 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:37.097435    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwcqk\" (UniqueName: \"kubernetes.io/projected/eac540e7-77f1-49ba-b3da-b875082bcaea-kube-api-access-fwcqk\") pod \"console-576bd68b6b-qpsch\" (UID: \"eac540e7-77f1-49ba-b3da-b875082bcaea\") " pod="openshift-console/console-576bd68b6b-qpsch"
Sep 08 13:18:37.097651 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:37.097493    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eac540e7-77f1-49ba-b3da-b875082bcaea-oauth-serving-cert\") pod \"console-576bd68b6b-qpsch\" (UID: \"eac540e7-77f1-49ba-b3da-b875082bcaea\") " pod="openshift-console/console-576bd68b6b-qpsch"
Sep 08 13:18:37.097651 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:37.097530    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eac540e7-77f1-49ba-b3da-b875082bcaea-console-oauth-config\") pod \"console-576bd68b6b-qpsch\" (UID: \"eac540e7-77f1-49ba-b3da-b875082bcaea\") " pod="openshift-console/console-576bd68b6b-qpsch"
Sep 08 13:18:37.097651 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:37.097568    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eac540e7-77f1-49ba-b3da-b875082bcaea-console-serving-cert\") pod \"console-576bd68b6b-qpsch\" (UID: \"eac540e7-77f1-49ba-b3da-b875082bcaea\") " pod="openshift-console/console-576bd68b6b-qpsch"
Sep 08 13:18:37.097651 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:37.097607    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eac540e7-77f1-49ba-b3da-b875082bcaea-service-ca\") pod \"console-576bd68b6b-qpsch\" (UID: \"eac540e7-77f1-49ba-b3da-b875082bcaea\") " pod="openshift-console/console-576bd68b6b-qpsch"
Sep 08 13:18:37.098388 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:37.098339    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eac540e7-77f1-49ba-b3da-b875082bcaea-console-config\") pod \"console-576bd68b6b-qpsch\" (UID: \"eac540e7-77f1-49ba-b3da-b875082bcaea\") " pod="openshift-console/console-576bd68b6b-qpsch"
Sep 08 13:18:37.098388 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:37.098379    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eac540e7-77f1-49ba-b3da-b875082bcaea-service-ca\") pod \"console-576bd68b6b-qpsch\" (UID: \"eac540e7-77f1-49ba-b3da-b875082bcaea\") " pod="openshift-console/console-576bd68b6b-qpsch"
Sep 08 13:18:37.098556 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:37.098466    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eac540e7-77f1-49ba-b3da-b875082bcaea-oauth-serving-cert\") pod \"console-576bd68b6b-qpsch\" (UID: \"eac540e7-77f1-49ba-b3da-b875082bcaea\") " pod="openshift-console/console-576bd68b6b-qpsch"
Sep 08 13:18:37.100555 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:37.100510    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eac540e7-77f1-49ba-b3da-b875082bcaea-console-oauth-config\") pod \"console-576bd68b6b-qpsch\" (UID: \"eac540e7-77f1-49ba-b3da-b875082bcaea\") " pod="openshift-console/console-576bd68b6b-qpsch"
Sep 08 13:18:37.100687 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:37.100562    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eac540e7-77f1-49ba-b3da-b875082bcaea-console-serving-cert\") pod \"console-576bd68b6b-qpsch\" (UID: \"eac540e7-77f1-49ba-b3da-b875082bcaea\") " pod="openshift-console/console-576bd68b6b-qpsch"
Sep 08 13:18:37.106857 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:37.106836    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwcqk\" (UniqueName: \"kubernetes.io/projected/eac540e7-77f1-49ba-b3da-b875082bcaea-kube-api-access-fwcqk\") pod \"console-576bd68b6b-qpsch\" (UID: \"eac540e7-77f1-49ba-b3da-b875082bcaea\") " pod="openshift-console/console-576bd68b6b-qpsch"
Sep 08 13:18:37.276456 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:37.276375    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-576bd68b6b-qpsch"
Sep 08 13:18:37.400323 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:37.400288    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5872b485-7dfa-45e1-8313-2393fc7361e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8nbzm\" (UID: \"5872b485-7dfa-45e1-8313-2393fc7361e6\") " pod="openshift-insights/insights-runtime-extractor-8nbzm"
Sep 08 13:18:37.402980 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:37.402950    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5872b485-7dfa-45e1-8313-2393fc7361e6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8nbzm\" (UID: \"5872b485-7dfa-45e1-8313-2393fc7361e6\") " pod="openshift-insights/insights-runtime-extractor-8nbzm"
Sep 08 13:18:37.561778 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:37.561689    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-576bd68b6b-qpsch"]
Sep 08 13:18:37.565284 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:18:37.565255    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeac540e7_77f1_49ba_b3da_b875082bcaea.slice/crio-bf51a99a9b4a74306a80107a1e2ae21ea68eafcbd4125330cc4bebb43706db77 WatchSource:0}: Error finding container bf51a99a9b4a74306a80107a1e2ae21ea68eafcbd4125330cc4bebb43706db77: Status 404 returned error can't find the container with id bf51a99a9b4a74306a80107a1e2ae21ea68eafcbd4125330cc4bebb43706db77
Sep 08 13:18:37.623438 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:37.623404    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-576bd68b6b-qpsch" event={"ID":"eac540e7-77f1-49ba-b3da-b875082bcaea","Type":"ContainerStarted","Data":"bf51a99a9b4a74306a80107a1e2ae21ea68eafcbd4125330cc4bebb43706db77"}
Sep 08 13:18:37.655163 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:37.655131    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-x5h6q\""
Sep 08 13:18:37.663255 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:37.663233    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8nbzm"
Sep 08 13:18:37.942153 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:37.942122    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8nbzm"]
Sep 08 13:18:37.945850 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:18:37.945820    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5872b485_7dfa_45e1_8313_2393fc7361e6.slice/crio-312dcf40d2d2067b3a2416038ad43befba964791282689e23f737868ff6d1117 WatchSource:0}: Error finding container 312dcf40d2d2067b3a2416038ad43befba964791282689e23f737868ff6d1117: Status 404 returned error can't find the container with id 312dcf40d2d2067b3a2416038ad43befba964791282689e23f737868ff6d1117
Sep 08 13:18:38.046342 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:38.046248    2558 patch_prober.go:28] interesting pod/image-registry-bcb89cbc9-g9vr7 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 08 13:18:38.046342 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:38.046304    2558 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7" podUID="494d9fb4-e6b1-4d95-a27d-194c4d4c6079" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:18:38.635226 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:38.635178    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8nbzm" event={"ID":"5872b485-7dfa-45e1-8313-2393fc7361e6","Type":"ContainerStarted","Data":"13a630d0f76649d34ec2508fef8c160a8d0df1da57ae3c1eb97b25eab24c798c"}
Sep 08 13:18:38.635416 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:38.635294    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8nbzm" event={"ID":"5872b485-7dfa-45e1-8313-2393fc7361e6","Type":"ContainerStarted","Data":"312dcf40d2d2067b3a2416038ad43befba964791282689e23f737868ff6d1117"}
Sep 08 13:18:39.560702 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:39.560656    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:18:39.923068 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:39.923031    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-x6c56"]
Sep 08 13:18:39.926148 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:39.926127    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-x6c56"
Sep 08 13:18:39.929389 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:39.929366    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 08 13:18:39.929511 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:39.929373    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-cc79s\""
Sep 08 13:18:39.936901 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:39.936875    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-x6c56"]
Sep 08 13:18:40.019800 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:40.019744    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/09b7c4f8-08f1-4730-93cb-e9672f1e09d3-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-x6c56\" (UID: \"09b7c4f8-08f1-4730-93cb-e9672f1e09d3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-x6c56"
Sep 08 13:18:40.120181 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:40.120144    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/09b7c4f8-08f1-4730-93cb-e9672f1e09d3-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-x6c56\" (UID: \"09b7c4f8-08f1-4730-93cb-e9672f1e09d3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-x6c56"
Sep 08 13:18:40.120332 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:40.120287    2558 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 08 13:18:40.120406 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:40.120372    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09b7c4f8-08f1-4730-93cb-e9672f1e09d3-tls-certificates podName:09b7c4f8-08f1-4730-93cb-e9672f1e09d3 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:40.620350594 +0000 UTC m=+184.072111512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/09b7c4f8-08f1-4730-93cb-e9672f1e09d3-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-x6c56" (UID: "09b7c4f8-08f1-4730-93cb-e9672f1e09d3") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 08 13:18:40.623204 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:40.623168    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/09b7c4f8-08f1-4730-93cb-e9672f1e09d3-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-x6c56\" (UID: \"09b7c4f8-08f1-4730-93cb-e9672f1e09d3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-x6c56"
Sep 08 13:18:40.626056 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:40.625996    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/09b7c4f8-08f1-4730-93cb-e9672f1e09d3-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-x6c56\" (UID: \"09b7c4f8-08f1-4730-93cb-e9672f1e09d3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-x6c56"
Sep 08 13:18:40.836951 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:40.836915    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-x6c56"
Sep 08 13:18:46.671660 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:46.671616    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8nbzm" event={"ID":"5872b485-7dfa-45e1-8313-2393fc7361e6","Type":"ContainerStarted","Data":"d616ee28a244107ec9f7abcf0f5e9bd6c7d0ee6479224d0d432b7f5c5960c772"}
Sep 08 13:18:47.477881 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:47.477794    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-x6c56"]
Sep 08 13:18:47.482256 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:18:47.482206    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b7c4f8_08f1_4730_93cb_e9672f1e09d3.slice/crio-4f536636056122b91fa4dd4bd84e70b80fe98a00818bc9b29e259a75a9ae5505 WatchSource:0}: Error finding container 4f536636056122b91fa4dd4bd84e70b80fe98a00818bc9b29e259a75a9ae5505: Status 404 returned error can't find the container with id 4f536636056122b91fa4dd4bd84e70b80fe98a00818bc9b29e259a75a9ae5505
Sep 08 13:18:47.677852 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:47.677765    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-x6c56" event={"ID":"09b7c4f8-08f1-4730-93cb-e9672f1e09d3","Type":"ContainerStarted","Data":"4f536636056122b91fa4dd4bd84e70b80fe98a00818bc9b29e259a75a9ae5505"}
Sep 08 13:18:47.680405 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:47.680369    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-hz955" event={"ID":"46b98f99-3985-4376-818e-269810605864","Type":"ContainerStarted","Data":"800d9c254dbc9918979caae950791b0e3803301e65bb4710628a853cce6f2169"}
Sep 08 13:18:47.680835 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:47.680641    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-hz955"
Sep 08 13:18:47.682900 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:47.682634    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-576bd68b6b-qpsch" event={"ID":"eac540e7-77f1-49ba-b3da-b875082bcaea","Type":"ContainerStarted","Data":"028522ec7914ccdcc004320d353dea0425fd7c2c36789e3da8993dd043cb125c"}
Sep 08 13:18:47.691331 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:47.691293    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-hz955"
Sep 08 13:18:47.700674 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:47.700394    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-hz955" podStartSLOduration=1.243348198 podStartE2EDuration="18.700380255s" totalImagesPullingTime="17.457032057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:18:47.698271789 +0000 UTC m=+191.150032725" watchObservedRunningTime="2026-09-08 13:18:47.700380255 +0000 UTC m=+191.152141186"
Sep 08 13:18:48.690664 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:48.690615    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8nbzm" event={"ID":"5872b485-7dfa-45e1-8313-2393fc7361e6","Type":"ContainerStarted","Data":"5bc4a9635a08a8219e6f911812f6265f9a4182db2789d596f1c1e7d4d6cac693"}
Sep 08 13:18:48.710734 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:48.710670    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8nbzm" podStartSLOduration=40.50534104 podStartE2EDuration="43.710648455s" totalImagesPullingTime="3.205307415s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:18:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:18:48.709515862 +0000 UTC m=+192.161276799" watchObservedRunningTime="2026-09-08 13:18:48.710648455 +0000 UTC m=+192.162409391"
Sep 08 13:18:48.711869 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:48.711785    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-576bd68b6b-qpsch" podStartSLOduration=3.062297623 podStartE2EDuration="12.711772067s" totalImagesPullingTime="9.649474444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:18:47.73814465 +0000 UTC m=+191.189905581" watchObservedRunningTime="2026-09-08 13:18:48.711772067 +0000 UTC m=+192.163533003"
Sep 08 13:18:50.700444 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:50.700385    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-x6c56" event={"ID":"09b7c4f8-08f1-4730-93cb-e9672f1e09d3","Type":"ContainerStarted","Data":"49a059049f10b620a2ca8083a851e16c76daf11051527b3758ac22a1bea6d436"}
Sep 08 13:18:50.700906 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:50.700822    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-x6c56"
Sep 08 13:18:50.707683 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:50.707654    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-x6c56"
Sep 08 13:18:50.721082 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:50.721027    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-x6c56" podStartSLOduration=9.611039018 podStartE2EDuration="11.721008953s" totalImagesPullingTime="2.109969935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:18:50.720161405 +0000 UTC m=+194.171922337" watchObservedRunningTime="2026-09-08 13:18:50.721008953 +0000 UTC m=+194.172769888"
Sep 08 13:18:51.526679 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:51.526642    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-bcb89cbc9-g9vr7"]
Sep 08 13:18:55.625582 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:55.625540    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-576bd68b6b-qpsch"]
Sep 08 13:18:56.363093 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.363052    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-p84l2"]
Sep 08 13:18:56.424898 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.424859    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p84l2"
Sep 08 13:18:56.427247 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.427221    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 08 13:18:56.427452 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.427430    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7sv8r\""
Sep 08 13:18:56.428130 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.428109    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 08 13:18:56.428130 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.428121    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 08 13:18:56.428313 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.428295    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 08 13:18:56.428373 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.428314    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 08 13:18:56.428746 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.428728    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 08 13:18:56.565999 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.565959    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e84ecd52-fe2b-4159-99e1-dd97713d9a65-sys\") pod \"node-exporter-p84l2\" (UID: \"e84ecd52-fe2b-4159-99e1-dd97713d9a65\") " pod="openshift-monitoring/node-exporter-p84l2"
Sep 08 13:18:56.565999 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.566003    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e84ecd52-fe2b-4159-99e1-dd97713d9a65-node-exporter-textfile\") pod \"node-exporter-p84l2\" (UID: \"e84ecd52-fe2b-4159-99e1-dd97713d9a65\") " pod="openshift-monitoring/node-exporter-p84l2"
Sep 08 13:18:56.566232 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.566035    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bx4s\" (UniqueName: \"kubernetes.io/projected/e84ecd52-fe2b-4159-99e1-dd97713d9a65-kube-api-access-5bx4s\") pod \"node-exporter-p84l2\" (UID: \"e84ecd52-fe2b-4159-99e1-dd97713d9a65\") " pod="openshift-monitoring/node-exporter-p84l2"
Sep 08 13:18:56.566232 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.566100    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e84ecd52-fe2b-4159-99e1-dd97713d9a65-node-exporter-accelerators-collector-config\") pod \"node-exporter-p84l2\" (UID: \"e84ecd52-fe2b-4159-99e1-dd97713d9a65\") " pod="openshift-monitoring/node-exporter-p84l2"
Sep 08 13:18:56.566232 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.566136    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e84ecd52-fe2b-4159-99e1-dd97713d9a65-node-exporter-wtmp\") pod \"node-exporter-p84l2\" (UID: \"e84ecd52-fe2b-4159-99e1-dd97713d9a65\") " pod="openshift-monitoring/node-exporter-p84l2"
Sep 08 13:18:56.566232 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.566218    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e84ecd52-fe2b-4159-99e1-dd97713d9a65-root\") pod \"node-exporter-p84l2\" (UID: \"e84ecd52-fe2b-4159-99e1-dd97713d9a65\") " pod="openshift-monitoring/node-exporter-p84l2"
Sep 08 13:18:56.566379 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.566245    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e84ecd52-fe2b-4159-99e1-dd97713d9a65-node-exporter-tls\") pod \"node-exporter-p84l2\" (UID: \"e84ecd52-fe2b-4159-99e1-dd97713d9a65\") " pod="openshift-monitoring/node-exporter-p84l2"
Sep 08 13:18:56.566379 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.566265    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e84ecd52-fe2b-4159-99e1-dd97713d9a65-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p84l2\" (UID: \"e84ecd52-fe2b-4159-99e1-dd97713d9a65\") " pod="openshift-monitoring/node-exporter-p84l2"
Sep 08 13:18:56.566379 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.566282    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e84ecd52-fe2b-4159-99e1-dd97713d9a65-metrics-client-ca\") pod \"node-exporter-p84l2\" (UID: \"e84ecd52-fe2b-4159-99e1-dd97713d9a65\") " pod="openshift-monitoring/node-exporter-p84l2"
Sep 08 13:18:56.666761 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.666672    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bx4s\" (UniqueName: \"kubernetes.io/projected/e84ecd52-fe2b-4159-99e1-dd97713d9a65-kube-api-access-5bx4s\") pod \"node-exporter-p84l2\" (UID: \"e84ecd52-fe2b-4159-99e1-dd97713d9a65\") " pod="openshift-monitoring/node-exporter-p84l2"
Sep 08 13:18:56.666761 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.666722    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e84ecd52-fe2b-4159-99e1-dd97713d9a65-node-exporter-accelerators-collector-config\") pod \"node-exporter-p84l2\" (UID: \"e84ecd52-fe2b-4159-99e1-dd97713d9a65\") " pod="openshift-monitoring/node-exporter-p84l2"
Sep 08 13:18:56.666761 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.666746    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e84ecd52-fe2b-4159-99e1-dd97713d9a65-node-exporter-wtmp\") pod \"node-exporter-p84l2\" (UID: \"e84ecd52-fe2b-4159-99e1-dd97713d9a65\") " pod="openshift-monitoring/node-exporter-p84l2"
Sep 08 13:18:56.667335 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.666787    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e84ecd52-fe2b-4159-99e1-dd97713d9a65-root\") pod \"node-exporter-p84l2\" (UID: \"e84ecd52-fe2b-4159-99e1-dd97713d9a65\") " pod="openshift-monitoring/node-exporter-p84l2"
Sep 08 13:18:56.667335 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.666802    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e84ecd52-fe2b-4159-99e1-dd97713d9a65-node-exporter-tls\") pod \"node-exporter-p84l2\" (UID: \"e84ecd52-fe2b-4159-99e1-dd97713d9a65\") " pod="openshift-monitoring/node-exporter-p84l2"
Sep 08 13:18:56.667335 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.666818    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e84ecd52-fe2b-4159-99e1-dd97713d9a65-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p84l2\" (UID: \"e84ecd52-fe2b-4159-99e1-dd97713d9a65\") " pod="openshift-monitoring/node-exporter-p84l2"
Sep 08 13:18:56.667335 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.666836    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e84ecd52-fe2b-4159-99e1-dd97713d9a65-metrics-client-ca\") pod \"node-exporter-p84l2\" (UID: \"e84ecd52-fe2b-4159-99e1-dd97713d9a65\") " pod="openshift-monitoring/node-exporter-p84l2"
Sep 08 13:18:56.667335 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.666858    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e84ecd52-fe2b-4159-99e1-dd97713d9a65-sys\") pod \"node-exporter-p84l2\" (UID: \"e84ecd52-fe2b-4159-99e1-dd97713d9a65\") " pod="openshift-monitoring/node-exporter-p84l2"
Sep 08 13:18:56.667335 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.666879    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e84ecd52-fe2b-4159-99e1-dd97713d9a65-root\") pod \"node-exporter-p84l2\" (UID: \"e84ecd52-fe2b-4159-99e1-dd97713d9a65\") " pod="openshift-monitoring/node-exporter-p84l2"
Sep 08 13:18:56.667335 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.666883    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e84ecd52-fe2b-4159-99e1-dd97713d9a65-node-exporter-textfile\") pod \"node-exporter-p84l2\" (UID: \"e84ecd52-fe2b-4159-99e1-dd97713d9a65\") " pod="openshift-monitoring/node-exporter-p84l2"
Sep 08 13:18:56.667335 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.666929    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e84ecd52-fe2b-4159-99e1-dd97713d9a65-sys\") pod \"node-exporter-p84l2\" (UID: \"e84ecd52-fe2b-4159-99e1-dd97713d9a65\") " pod="openshift-monitoring/node-exporter-p84l2"
Sep 08 13:18:56.667335 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.667287    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e84ecd52-fe2b-4159-99e1-dd97713d9a65-node-exporter-wtmp\") pod \"node-exporter-p84l2\" (UID: \"e84ecd52-fe2b-4159-99e1-dd97713d9a65\") " pod="openshift-monitoring/node-exporter-p84l2"
Sep 08 13:18:56.667335 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.667314    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e84ecd52-fe2b-4159-99e1-dd97713d9a65-node-exporter-textfile\") pod \"node-exporter-p84l2\" (UID: \"e84ecd52-fe2b-4159-99e1-dd97713d9a65\") " pod="openshift-monitoring/node-exporter-p84l2"
Sep 08 13:18:56.667725 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.667375    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e84ecd52-fe2b-4159-99e1-dd97713d9a65-node-exporter-accelerators-collector-config\") pod \"node-exporter-p84l2\" (UID: \"e84ecd52-fe2b-4159-99e1-dd97713d9a65\") " pod="openshift-monitoring/node-exporter-p84l2"
Sep 08 13:18:56.667725 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.667509    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e84ecd52-fe2b-4159-99e1-dd97713d9a65-metrics-client-ca\") pod \"node-exporter-p84l2\" (UID: \"e84ecd52-fe2b-4159-99e1-dd97713d9a65\") " pod="openshift-monitoring/node-exporter-p84l2"
Sep 08 13:18:56.669302 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.669272    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e84ecd52-fe2b-4159-99e1-dd97713d9a65-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p84l2\" (UID: \"e84ecd52-fe2b-4159-99e1-dd97713d9a65\") " pod="openshift-monitoring/node-exporter-p84l2"
Sep 08 13:18:56.669566 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.669299    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e84ecd52-fe2b-4159-99e1-dd97713d9a65-node-exporter-tls\") pod \"node-exporter-p84l2\" (UID: \"e84ecd52-fe2b-4159-99e1-dd97713d9a65\") " pod="openshift-monitoring/node-exporter-p84l2"
Sep 08 13:18:56.675035 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.675009    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bx4s\" (UniqueName: \"kubernetes.io/projected/e84ecd52-fe2b-4159-99e1-dd97713d9a65-kube-api-access-5bx4s\") pod \"node-exporter-p84l2\" (UID: \"e84ecd52-fe2b-4159-99e1-dd97713d9a65\") " pod="openshift-monitoring/node-exporter-p84l2"
Sep 08 13:18:56.737236 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:56.737204    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p84l2"
Sep 08 13:18:56.778471 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:18:56.778437    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode84ecd52_fe2b_4159_99e1_dd97713d9a65.slice/crio-00285720ccf11f52b195d288d7281c372962d5760c68a14610fee8d98e90dbbf WatchSource:0}: Error finding container 00285720ccf11f52b195d288d7281c372962d5760c68a14610fee8d98e90dbbf: Status 404 returned error can't find the container with id 00285720ccf11f52b195d288d7281c372962d5760c68a14610fee8d98e90dbbf
Sep 08 13:18:57.276858 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.276821    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-576bd68b6b-qpsch"
Sep 08 13:18:57.419678 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.419647    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 13:18:57.452401 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.452366    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 13:18:57.452550 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.452487    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.455084 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.455061    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 08 13:18:57.455223 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.455066    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 08 13:18:57.455223 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.455066    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 08 13:18:57.455343 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.455066    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 08 13:18:57.455483 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.455459    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 08 13:18:57.455665 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.455504    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 08 13:18:57.455777 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.455525    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 08 13:18:57.455833 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.455531    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xtcsh\""
Sep 08 13:18:57.455881 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.455621    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 08 13:18:57.455924 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.455652    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 08 13:18:57.574964 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.574873    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ab57a63e-5ff7-4bdb-a5b9-70947b872230-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.574964 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.574927    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab57a63e-5ff7-4bdb-a5b9-70947b872230-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.574964 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.574945    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ab57a63e-5ff7-4bdb-a5b9-70947b872230-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.575185 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.575015    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ab57a63e-5ff7-4bdb-a5b9-70947b872230-web-config\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.575185 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.575061    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcspk\" (UniqueName: \"kubernetes.io/projected/ab57a63e-5ff7-4bdb-a5b9-70947b872230-kube-api-access-jcspk\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.575185 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.575080    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ab57a63e-5ff7-4bdb-a5b9-70947b872230-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.575185 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.575097    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ab57a63e-5ff7-4bdb-a5b9-70947b872230-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.575185 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.575125    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ab57a63e-5ff7-4bdb-a5b9-70947b872230-config-out\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.575185 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.575148    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ab57a63e-5ff7-4bdb-a5b9-70947b872230-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.575440 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.575190    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab57a63e-5ff7-4bdb-a5b9-70947b872230-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.575440 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.575232    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ab57a63e-5ff7-4bdb-a5b9-70947b872230-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.575440 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.575283    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ab57a63e-5ff7-4bdb-a5b9-70947b872230-config-volume\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.575440 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.575321    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ab57a63e-5ff7-4bdb-a5b9-70947b872230-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.676055 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.676019    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ab57a63e-5ff7-4bdb-a5b9-70947b872230-web-config\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.676055 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.676056    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcspk\" (UniqueName: \"kubernetes.io/projected/ab57a63e-5ff7-4bdb-a5b9-70947b872230-kube-api-access-jcspk\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.676581 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.676075    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ab57a63e-5ff7-4bdb-a5b9-70947b872230-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.676581 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.676096    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ab57a63e-5ff7-4bdb-a5b9-70947b872230-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.676581 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.676126    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ab57a63e-5ff7-4bdb-a5b9-70947b872230-config-out\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.676581 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.676149    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ab57a63e-5ff7-4bdb-a5b9-70947b872230-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.676581 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.676181    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab57a63e-5ff7-4bdb-a5b9-70947b872230-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.676581 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:57.676207    2558 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 08 13:18:57.676581 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.676250    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ab57a63e-5ff7-4bdb-a5b9-70947b872230-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.676581 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:18:57.676279    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab57a63e-5ff7-4bdb-a5b9-70947b872230-secret-alertmanager-main-tls podName:ab57a63e-5ff7-4bdb-a5b9-70947b872230 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:58.176258723 +0000 UTC m=+201.628019632 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/ab57a63e-5ff7-4bdb-a5b9-70947b872230-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "ab57a63e-5ff7-4bdb-a5b9-70947b872230") : secret "alertmanager-main-tls" not found
Sep 08 13:18:57.676581 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.676325    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ab57a63e-5ff7-4bdb-a5b9-70947b872230-config-volume\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.676581 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.676354    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ab57a63e-5ff7-4bdb-a5b9-70947b872230-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.676581 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.676404    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ab57a63e-5ff7-4bdb-a5b9-70947b872230-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.676581 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.676450    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab57a63e-5ff7-4bdb-a5b9-70947b872230-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.676581 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.676475    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ab57a63e-5ff7-4bdb-a5b9-70947b872230-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.676581 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.676560    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ab57a63e-5ff7-4bdb-a5b9-70947b872230-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.677283 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.676995    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab57a63e-5ff7-4bdb-a5b9-70947b872230-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.677987 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.677906    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab57a63e-5ff7-4bdb-a5b9-70947b872230-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.679415 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.679186    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ab57a63e-5ff7-4bdb-a5b9-70947b872230-web-config\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.679415 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.679243    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ab57a63e-5ff7-4bdb-a5b9-70947b872230-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.679415 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.679366    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ab57a63e-5ff7-4bdb-a5b9-70947b872230-config-out\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.679415 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.679369    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ab57a63e-5ff7-4bdb-a5b9-70947b872230-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.679706 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.679568    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ab57a63e-5ff7-4bdb-a5b9-70947b872230-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.680066 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.680047    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ab57a63e-5ff7-4bdb-a5b9-70947b872230-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.680264 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.680247    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ab57a63e-5ff7-4bdb-a5b9-70947b872230-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.680378 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.680362    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ab57a63e-5ff7-4bdb-a5b9-70947b872230-config-volume\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.688848 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.688820    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcspk\" (UniqueName: \"kubernetes.io/projected/ab57a63e-5ff7-4bdb-a5b9-70947b872230-kube-api-access-jcspk\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:57.731119 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:57.731084    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p84l2" event={"ID":"e84ecd52-fe2b-4159-99e1-dd97713d9a65","Type":"ContainerStarted","Data":"00285720ccf11f52b195d288d7281c372962d5760c68a14610fee8d98e90dbbf"}
Sep 08 13:18:58.180986 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.180941    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ab57a63e-5ff7-4bdb-a5b9-70947b872230-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:58.183368 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.183335    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ab57a63e-5ff7-4bdb-a5b9-70947b872230-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ab57a63e-5ff7-4bdb-a5b9-70947b872230\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:58.340858 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.340824    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-cd4f7799f-bnz9h"]
Sep 08 13:18:58.357240 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.357210    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-cd4f7799f-bnz9h"]
Sep 08 13:18:58.357412 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.357330    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h"
Sep 08 13:18:58.359938 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.359908    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 08 13:18:58.360082 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.359946    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-248bc\""
Sep 08 13:18:58.360082 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.359908    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 08 13:18:58.360082 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.360034    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 08 13:18:58.360082 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.359908    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 08 13:18:58.360082 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.359908    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 08 13:18:58.360338 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.360181    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-cnjqknujs98aa\""
Sep 08 13:18:58.362713 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.362695    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:18:58.484325 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.484241    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc05686f-7483-4834-afcc-f89ce4721ccc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-cd4f7799f-bnz9h\" (UID: \"cc05686f-7483-4834-afcc-f89ce4721ccc\") " pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h"
Sep 08 13:18:58.484325 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.484304    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc05686f-7483-4834-afcc-f89ce4721ccc-metrics-client-ca\") pod \"thanos-querier-cd4f7799f-bnz9h\" (UID: \"cc05686f-7483-4834-afcc-f89ce4721ccc\") " pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h"
Sep 08 13:18:58.484508 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.484372    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc05686f-7483-4834-afcc-f89ce4721ccc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-cd4f7799f-bnz9h\" (UID: \"cc05686f-7483-4834-afcc-f89ce4721ccc\") " pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h"
Sep 08 13:18:58.484508 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.484441    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6d65\" (UniqueName: \"kubernetes.io/projected/cc05686f-7483-4834-afcc-f89ce4721ccc-kube-api-access-c6d65\") pod \"thanos-querier-cd4f7799f-bnz9h\" (UID: \"cc05686f-7483-4834-afcc-f89ce4721ccc\") " pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h"
Sep 08 13:18:58.484508 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.484462    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cc05686f-7483-4834-afcc-f89ce4721ccc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-cd4f7799f-bnz9h\" (UID: \"cc05686f-7483-4834-afcc-f89ce4721ccc\") " pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h"
Sep 08 13:18:58.484629 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.484533    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cc05686f-7483-4834-afcc-f89ce4721ccc-secret-thanos-querier-tls\") pod \"thanos-querier-cd4f7799f-bnz9h\" (UID: \"cc05686f-7483-4834-afcc-f89ce4721ccc\") " pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h"
Sep 08 13:18:58.484629 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.484568    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc05686f-7483-4834-afcc-f89ce4721ccc-secret-grpc-tls\") pod \"thanos-querier-cd4f7799f-bnz9h\" (UID: \"cc05686f-7483-4834-afcc-f89ce4721ccc\") " pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h"
Sep 08 13:18:58.484629 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.484587    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cc05686f-7483-4834-afcc-f89ce4721ccc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-cd4f7799f-bnz9h\" (UID: \"cc05686f-7483-4834-afcc-f89ce4721ccc\") " pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h"
Sep 08 13:18:58.585503 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.585464    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6d65\" (UniqueName: \"kubernetes.io/projected/cc05686f-7483-4834-afcc-f89ce4721ccc-kube-api-access-c6d65\") pod \"thanos-querier-cd4f7799f-bnz9h\" (UID: \"cc05686f-7483-4834-afcc-f89ce4721ccc\") " pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h"
Sep 08 13:18:58.585503 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.585505    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cc05686f-7483-4834-afcc-f89ce4721ccc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-cd4f7799f-bnz9h\" (UID: \"cc05686f-7483-4834-afcc-f89ce4721ccc\") " pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h"
Sep 08 13:18:58.585774 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.585532    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cc05686f-7483-4834-afcc-f89ce4721ccc-secret-thanos-querier-tls\") pod \"thanos-querier-cd4f7799f-bnz9h\" (UID: \"cc05686f-7483-4834-afcc-f89ce4721ccc\") " pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h"
Sep 08 13:18:58.585774 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.585615    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc05686f-7483-4834-afcc-f89ce4721ccc-secret-grpc-tls\") pod \"thanos-querier-cd4f7799f-bnz9h\" (UID: \"cc05686f-7483-4834-afcc-f89ce4721ccc\") " pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h"
Sep 08 13:18:58.585774 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.585660    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cc05686f-7483-4834-afcc-f89ce4721ccc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-cd4f7799f-bnz9h\" (UID: \"cc05686f-7483-4834-afcc-f89ce4721ccc\") " pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h"
Sep 08 13:18:58.585774 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.585716    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc05686f-7483-4834-afcc-f89ce4721ccc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-cd4f7799f-bnz9h\" (UID: \"cc05686f-7483-4834-afcc-f89ce4721ccc\") " pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h"
Sep 08 13:18:58.585979 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.585786    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc05686f-7483-4834-afcc-f89ce4721ccc-metrics-client-ca\") pod \"thanos-querier-cd4f7799f-bnz9h\" (UID: \"cc05686f-7483-4834-afcc-f89ce4721ccc\") " pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h"
Sep 08 13:18:58.585979 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.585819    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc05686f-7483-4834-afcc-f89ce4721ccc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-cd4f7799f-bnz9h\" (UID: \"cc05686f-7483-4834-afcc-f89ce4721ccc\") " pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h"
Sep 08 13:18:58.586535 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.586505    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc05686f-7483-4834-afcc-f89ce4721ccc-metrics-client-ca\") pod \"thanos-querier-cd4f7799f-bnz9h\" (UID: \"cc05686f-7483-4834-afcc-f89ce4721ccc\") " pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h"
Sep 08 13:18:58.588252 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.588222    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cc05686f-7483-4834-afcc-f89ce4721ccc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-cd4f7799f-bnz9h\" (UID: \"cc05686f-7483-4834-afcc-f89ce4721ccc\") " pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h"
Sep 08 13:18:58.588405 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.588381    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cc05686f-7483-4834-afcc-f89ce4721ccc-secret-thanos-querier-tls\") pod \"thanos-querier-cd4f7799f-bnz9h\" (UID: \"cc05686f-7483-4834-afcc-f89ce4721ccc\") " pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h"
Sep 08 13:18:58.588473 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.588458    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cc05686f-7483-4834-afcc-f89ce4721ccc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-cd4f7799f-bnz9h\" (UID: \"cc05686f-7483-4834-afcc-f89ce4721ccc\") " pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h"
Sep 08 13:18:58.588538 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.588518    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc05686f-7483-4834-afcc-f89ce4721ccc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-cd4f7799f-bnz9h\" (UID: \"cc05686f-7483-4834-afcc-f89ce4721ccc\") " pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h"
Sep 08 13:18:58.588704 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.588687    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc05686f-7483-4834-afcc-f89ce4721ccc-secret-grpc-tls\") pod \"thanos-querier-cd4f7799f-bnz9h\" (UID: \"cc05686f-7483-4834-afcc-f89ce4721ccc\") " pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h"
Sep 08 13:18:58.588781 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.588765    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc05686f-7483-4834-afcc-f89ce4721ccc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-cd4f7799f-bnz9h\" (UID: \"cc05686f-7483-4834-afcc-f89ce4721ccc\") " pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h"
Sep 08 13:18:58.596755 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.596731    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6d65\" (UniqueName: \"kubernetes.io/projected/cc05686f-7483-4834-afcc-f89ce4721ccc-kube-api-access-c6d65\") pod \"thanos-querier-cd4f7799f-bnz9h\" (UID: \"cc05686f-7483-4834-afcc-f89ce4721ccc\") " pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h"
Sep 08 13:18:58.660980 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.660926    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 13:18:58.663297 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:18:58.663274    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab57a63e_5ff7_4bdb_a5b9_70947b872230.slice/crio-e367d0781b240ec1c9943975c664b406979ebf628505a8748f75c81c1ce54ca4 WatchSource:0}: Error finding container e367d0781b240ec1c9943975c664b406979ebf628505a8748f75c81c1ce54ca4: Status 404 returned error can't find the container with id e367d0781b240ec1c9943975c664b406979ebf628505a8748f75c81c1ce54ca4
Sep 08 13:18:58.666820 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.666797    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h"
Sep 08 13:18:58.736303 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.736219    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ab57a63e-5ff7-4bdb-a5b9-70947b872230","Type":"ContainerStarted","Data":"e367d0781b240ec1c9943975c664b406979ebf628505a8748f75c81c1ce54ca4"}
Sep 08 13:18:58.973820 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:58.973785    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-cd4f7799f-bnz9h"]
Sep 08 13:18:58.977467 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:18:58.977425    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc05686f_7483_4834_afcc_f89ce4721ccc.slice/crio-7e7c6b722301b0617edd00afa06b18beb6adaef59f722ed30442597eef7e4d43 WatchSource:0}: Error finding container 7e7c6b722301b0617edd00afa06b18beb6adaef59f722ed30442597eef7e4d43: Status 404 returned error can't find the container with id 7e7c6b722301b0617edd00afa06b18beb6adaef59f722ed30442597eef7e4d43
Sep 08 13:18:59.744801 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:18:59.744759    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h" event={"ID":"cc05686f-7483-4834-afcc-f89ce4721ccc","Type":"ContainerStarted","Data":"7e7c6b722301b0617edd00afa06b18beb6adaef59f722ed30442597eef7e4d43"}
Sep 08 13:19:00.751550 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:00.751515    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ab57a63e-5ff7-4bdb-a5b9-70947b872230","Type":"ContainerStarted","Data":"8d78ef4aa27612edec7b12d20a4e8e4e039f3d4f052756949671d921f1966502"}
Sep 08 13:19:01.160917 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.160874    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-jxhsg"]
Sep 08 13:19:01.178730 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.178702    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-jxhsg"]
Sep 08 13:19:01.178895 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.178819    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-jxhsg"
Sep 08 13:19:01.181812 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.181707    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 08 13:19:01.181948 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.181812    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-gr844\""
Sep 08 13:19:01.312618 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.312569    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5nvc\" (UniqueName: \"kubernetes.io/projected/0686fffb-a8a9-459f-878e-e434089eaf3d-kube-api-access-p5nvc\") pod \"monitoring-plugin-85bbbbbf4-jxhsg\" (UID: \"0686fffb-a8a9-459f-878e-e434089eaf3d\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-jxhsg"
Sep 08 13:19:01.312729 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.312640    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0686fffb-a8a9-459f-878e-e434089eaf3d-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-jxhsg\" (UID: \"0686fffb-a8a9-459f-878e-e434089eaf3d\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-jxhsg"
Sep 08 13:19:01.414009 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.413917    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0686fffb-a8a9-459f-878e-e434089eaf3d-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-jxhsg\" (UID: \"0686fffb-a8a9-459f-878e-e434089eaf3d\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-jxhsg"
Sep 08 13:19:01.414189 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.414050    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5nvc\" (UniqueName: \"kubernetes.io/projected/0686fffb-a8a9-459f-878e-e434089eaf3d-kube-api-access-p5nvc\") pod \"monitoring-plugin-85bbbbbf4-jxhsg\" (UID: \"0686fffb-a8a9-459f-878e-e434089eaf3d\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-jxhsg"
Sep 08 13:19:01.414189 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:19:01.414084    2558 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 08 13:19:01.414189 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:19:01.414161    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0686fffb-a8a9-459f-878e-e434089eaf3d-monitoring-plugin-cert podName:0686fffb-a8a9-459f-878e-e434089eaf3d nodeName:}" failed. No retries permitted until 2026-09-08 13:19:01.914144475 +0000 UTC m=+205.365905389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/0686fffb-a8a9-459f-878e-e434089eaf3d-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-jxhsg" (UID: "0686fffb-a8a9-459f-878e-e434089eaf3d") : secret "monitoring-plugin-cert" not found
Sep 08 13:19:01.423181 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.423155    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5nvc\" (UniqueName: \"kubernetes.io/projected/0686fffb-a8a9-459f-878e-e434089eaf3d-kube-api-access-p5nvc\") pod \"monitoring-plugin-85bbbbbf4-jxhsg\" (UID: \"0686fffb-a8a9-459f-878e-e434089eaf3d\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-jxhsg"
Sep 08 13:19:01.584161 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.583432    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-787d46f89f-pgss5"]
Sep 08 13:19:01.607798 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.607767    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-787d46f89f-pgss5"]
Sep 08 13:19:01.607957 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.607852    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-787d46f89f-pgss5"
Sep 08 13:19:01.610559 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.610538    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-b942j\""
Sep 08 13:19:01.610907 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.610623    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 08 13:19:01.610907 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.610648    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 08 13:19:01.610907 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.610664    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 08 13:19:01.610907 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.610672    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 08 13:19:01.610907 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.610775    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 08 13:19:01.616553 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.616510    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 08 13:19:01.716489 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.716404    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f553b081-188a-47fa-9cac-a798cef2d48f-serving-certs-ca-bundle\") pod \"telemeter-client-787d46f89f-pgss5\" (UID: \"f553b081-188a-47fa-9cac-a798cef2d48f\") " pod="openshift-monitoring/telemeter-client-787d46f89f-pgss5"
Sep 08 13:19:01.716489 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.716442    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f553b081-188a-47fa-9cac-a798cef2d48f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-787d46f89f-pgss5\" (UID: \"f553b081-188a-47fa-9cac-a798cef2d48f\") " pod="openshift-monitoring/telemeter-client-787d46f89f-pgss5"
Sep 08 13:19:01.716489 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.716465    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vbnl\" (UniqueName: \"kubernetes.io/projected/f553b081-188a-47fa-9cac-a798cef2d48f-kube-api-access-5vbnl\") pod \"telemeter-client-787d46f89f-pgss5\" (UID: \"f553b081-188a-47fa-9cac-a798cef2d48f\") " pod="openshift-monitoring/telemeter-client-787d46f89f-pgss5"
Sep 08 13:19:01.716820 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.716524    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f553b081-188a-47fa-9cac-a798cef2d48f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-787d46f89f-pgss5\" (UID: \"f553b081-188a-47fa-9cac-a798cef2d48f\") " pod="openshift-monitoring/telemeter-client-787d46f89f-pgss5"
Sep 08 13:19:01.716820 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.716554    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f553b081-188a-47fa-9cac-a798cef2d48f-secret-telemeter-client\") pod \"telemeter-client-787d46f89f-pgss5\" (UID: \"f553b081-188a-47fa-9cac-a798cef2d48f\") " pod="openshift-monitoring/telemeter-client-787d46f89f-pgss5"
Sep 08 13:19:01.716820 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.716572    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f553b081-188a-47fa-9cac-a798cef2d48f-telemeter-client-tls\") pod \"telemeter-client-787d46f89f-pgss5\" (UID: \"f553b081-188a-47fa-9cac-a798cef2d48f\") " pod="openshift-monitoring/telemeter-client-787d46f89f-pgss5"
Sep 08 13:19:01.716820 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.716612    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f553b081-188a-47fa-9cac-a798cef2d48f-federate-client-tls\") pod \"telemeter-client-787d46f89f-pgss5\" (UID: \"f553b081-188a-47fa-9cac-a798cef2d48f\") " pod="openshift-monitoring/telemeter-client-787d46f89f-pgss5"
Sep 08 13:19:01.716820 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.716708    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f553b081-188a-47fa-9cac-a798cef2d48f-metrics-client-ca\") pod \"telemeter-client-787d46f89f-pgss5\" (UID: \"f553b081-188a-47fa-9cac-a798cef2d48f\") " pod="openshift-monitoring/telemeter-client-787d46f89f-pgss5"
Sep 08 13:19:01.756670 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.756633    2558 generic.go:356] "Generic (PLEG): container finished" podID="ab57a63e-5ff7-4bdb-a5b9-70947b872230" containerID="8d78ef4aa27612edec7b12d20a4e8e4e039f3d4f052756949671d921f1966502" exitCode=0
Sep 08 13:19:01.757144 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.756694    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ab57a63e-5ff7-4bdb-a5b9-70947b872230","Type":"ContainerDied","Data":"8d78ef4aa27612edec7b12d20a4e8e4e039f3d4f052756949671d921f1966502"}
Sep 08 13:19:01.817311 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.817280    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f553b081-188a-47fa-9cac-a798cef2d48f-metrics-client-ca\") pod \"telemeter-client-787d46f89f-pgss5\" (UID: \"f553b081-188a-47fa-9cac-a798cef2d48f\") " pod="openshift-monitoring/telemeter-client-787d46f89f-pgss5"
Sep 08 13:19:01.817480 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.817345    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f553b081-188a-47fa-9cac-a798cef2d48f-serving-certs-ca-bundle\") pod \"telemeter-client-787d46f89f-pgss5\" (UID: \"f553b081-188a-47fa-9cac-a798cef2d48f\") " pod="openshift-monitoring/telemeter-client-787d46f89f-pgss5"
Sep 08 13:19:01.817480 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.817372    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f553b081-188a-47fa-9cac-a798cef2d48f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-787d46f89f-pgss5\" (UID: \"f553b081-188a-47fa-9cac-a798cef2d48f\") " pod="openshift-monitoring/telemeter-client-787d46f89f-pgss5"
Sep 08 13:19:01.817480 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.817395    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vbnl\" (UniqueName: \"kubernetes.io/projected/f553b081-188a-47fa-9cac-a798cef2d48f-kube-api-access-5vbnl\") pod \"telemeter-client-787d46f89f-pgss5\" (UID: \"f553b081-188a-47fa-9cac-a798cef2d48f\") " pod="openshift-monitoring/telemeter-client-787d46f89f-pgss5"
Sep 08 13:19:01.817480 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.817417    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f553b081-188a-47fa-9cac-a798cef2d48f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-787d46f89f-pgss5\" (UID: \"f553b081-188a-47fa-9cac-a798cef2d48f\") " pod="openshift-monitoring/telemeter-client-787d46f89f-pgss5"
Sep 08 13:19:01.817480 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.817445    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f553b081-188a-47fa-9cac-a798cef2d48f-secret-telemeter-client\") pod \"telemeter-client-787d46f89f-pgss5\" (UID: \"f553b081-188a-47fa-9cac-a798cef2d48f\") " pod="openshift-monitoring/telemeter-client-787d46f89f-pgss5"
Sep 08 13:19:01.817480 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.817469    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f553b081-188a-47fa-9cac-a798cef2d48f-telemeter-client-tls\") pod \"telemeter-client-787d46f89f-pgss5\" (UID: \"f553b081-188a-47fa-9cac-a798cef2d48f\") " pod="openshift-monitoring/telemeter-client-787d46f89f-pgss5"
Sep 08 13:19:01.817786 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.817560    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f553b081-188a-47fa-9cac-a798cef2d48f-federate-client-tls\") pod \"telemeter-client-787d46f89f-pgss5\" (UID: \"f553b081-188a-47fa-9cac-a798cef2d48f\") " pod="openshift-monitoring/telemeter-client-787d46f89f-pgss5"
Sep 08 13:19:01.818150 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.818117    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f553b081-188a-47fa-9cac-a798cef2d48f-metrics-client-ca\") pod \"telemeter-client-787d46f89f-pgss5\" (UID: \"f553b081-188a-47fa-9cac-a798cef2d48f\") " pod="openshift-monitoring/telemeter-client-787d46f89f-pgss5"
Sep 08 13:19:01.818150 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.818135    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f553b081-188a-47fa-9cac-a798cef2d48f-serving-certs-ca-bundle\") pod \"telemeter-client-787d46f89f-pgss5\" (UID: \"f553b081-188a-47fa-9cac-a798cef2d48f\") " pod="openshift-monitoring/telemeter-client-787d46f89f-pgss5"
Sep 08 13:19:01.818532 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.818504    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f553b081-188a-47fa-9cac-a798cef2d48f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-787d46f89f-pgss5\" (UID: \"f553b081-188a-47fa-9cac-a798cef2d48f\") " pod="openshift-monitoring/telemeter-client-787d46f89f-pgss5"
Sep 08 13:19:01.820252 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.820227    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f553b081-188a-47fa-9cac-a798cef2d48f-secret-telemeter-client\") pod \"telemeter-client-787d46f89f-pgss5\" (UID: \"f553b081-188a-47fa-9cac-a798cef2d48f\") " pod="openshift-monitoring/telemeter-client-787d46f89f-pgss5"
Sep 08 13:19:01.820342 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.820259    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f553b081-188a-47fa-9cac-a798cef2d48f-telemeter-client-tls\") pod \"telemeter-client-787d46f89f-pgss5\" (UID: \"f553b081-188a-47fa-9cac-a798cef2d48f\") " pod="openshift-monitoring/telemeter-client-787d46f89f-pgss5"
Sep 08 13:19:01.820342 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.820259    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f553b081-188a-47fa-9cac-a798cef2d48f-federate-client-tls\") pod \"telemeter-client-787d46f89f-pgss5\" (UID: \"f553b081-188a-47fa-9cac-a798cef2d48f\") " pod="openshift-monitoring/telemeter-client-787d46f89f-pgss5"
Sep 08 13:19:01.820342 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.820230    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f553b081-188a-47fa-9cac-a798cef2d48f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-787d46f89f-pgss5\" (UID: \"f553b081-188a-47fa-9cac-a798cef2d48f\") " pod="openshift-monitoring/telemeter-client-787d46f89f-pgss5"
Sep 08 13:19:01.830171 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.830148    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vbnl\" (UniqueName: \"kubernetes.io/projected/f553b081-188a-47fa-9cac-a798cef2d48f-kube-api-access-5vbnl\") pod \"telemeter-client-787d46f89f-pgss5\" (UID: \"f553b081-188a-47fa-9cac-a798cef2d48f\") " pod="openshift-monitoring/telemeter-client-787d46f89f-pgss5"
Sep 08 13:19:01.918713 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.918677    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-787d46f89f-pgss5"
Sep 08 13:19:01.918870 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:01.918843    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0686fffb-a8a9-459f-878e-e434089eaf3d-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-jxhsg\" (UID: \"0686fffb-a8a9-459f-878e-e434089eaf3d\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-jxhsg"
Sep 08 13:19:01.919070 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:19:01.919051    2558 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 08 13:19:01.919141 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:19:01.919106    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0686fffb-a8a9-459f-878e-e434089eaf3d-monitoring-plugin-cert podName:0686fffb-a8a9-459f-878e-e434089eaf3d nodeName:}" failed. No retries permitted until 2026-09-08 13:19:02.919089261 +0000 UTC m=+206.370850170 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/0686fffb-a8a9-459f-878e-e434089eaf3d-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-jxhsg" (UID: "0686fffb-a8a9-459f-878e-e434089eaf3d") : secret "monitoring-plugin-cert" not found
Sep 08 13:19:02.231270 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:02.231218    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-787d46f89f-pgss5"]
Sep 08 13:19:02.238668 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:19:02.238638    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf553b081_188a_47fa_9cac_a798cef2d48f.slice/crio-6b1d1ac34c17eacaefd56e3ff87cfd4b63d46a19d518a470e049dab7ec9281af WatchSource:0}: Error finding container 6b1d1ac34c17eacaefd56e3ff87cfd4b63d46a19d518a470e049dab7ec9281af: Status 404 returned error can't find the container with id 6b1d1ac34c17eacaefd56e3ff87cfd4b63d46a19d518a470e049dab7ec9281af
Sep 08 13:19:02.761537 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:02.761498    2558 generic.go:356] "Generic (PLEG): container finished" podID="e84ecd52-fe2b-4159-99e1-dd97713d9a65" containerID="9937548ce6de44a6234b4bd1142c184288b36d7e3b0afc067a713ae9b5dd9c6b" exitCode=0
Sep 08 13:19:02.762049 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:02.761610    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p84l2" event={"ID":"e84ecd52-fe2b-4159-99e1-dd97713d9a65","Type":"ContainerDied","Data":"9937548ce6de44a6234b4bd1142c184288b36d7e3b0afc067a713ae9b5dd9c6b"}
Sep 08 13:19:02.763916 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:02.763889    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h" event={"ID":"cc05686f-7483-4834-afcc-f89ce4721ccc","Type":"ContainerStarted","Data":"7afe8a085ee631948f5ad358a47d00f02060495ab1968ef348f2f48c8da0a86a"}
Sep 08 13:19:02.764034 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:02.763967    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h" event={"ID":"cc05686f-7483-4834-afcc-f89ce4721ccc","Type":"ContainerStarted","Data":"a5d19f6880e4db3197470702f14b41fc16be207ed8e11aac7dad43df745b2ab1"}
Sep 08 13:19:02.764034 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:02.764008    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h" event={"ID":"cc05686f-7483-4834-afcc-f89ce4721ccc","Type":"ContainerStarted","Data":"a740fd7b1c92b747e29bb54c54503f945abfcf6fd8be3057dbbd7edb2ee21fbc"}
Sep 08 13:19:02.765006 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:02.764988    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-787d46f89f-pgss5" event={"ID":"f553b081-188a-47fa-9cac-a798cef2d48f","Type":"ContainerStarted","Data":"6b1d1ac34c17eacaefd56e3ff87cfd4b63d46a19d518a470e049dab7ec9281af"}
Sep 08 13:19:02.929949 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:02.929922    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0686fffb-a8a9-459f-878e-e434089eaf3d-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-jxhsg\" (UID: \"0686fffb-a8a9-459f-878e-e434089eaf3d\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-jxhsg"
Sep 08 13:19:02.932414 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:02.932389    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0686fffb-a8a9-459f-878e-e434089eaf3d-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-jxhsg\" (UID: \"0686fffb-a8a9-459f-878e-e434089eaf3d\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-jxhsg"
Sep 08 13:19:02.991728 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:02.991705    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-jxhsg"
Sep 08 13:19:03.286192 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:03.286037    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-jxhsg"]
Sep 08 13:19:03.393043 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:19:03.393010    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0686fffb_a8a9_459f_878e_e434089eaf3d.slice/crio-c3f521173459cb0813896fc6b8eed652d65c7b29616f82724434331db2559e5a WatchSource:0}: Error finding container c3f521173459cb0813896fc6b8eed652d65c7b29616f82724434331db2559e5a: Status 404 returned error can't find the container with id c3f521173459cb0813896fc6b8eed652d65c7b29616f82724434331db2559e5a
Sep 08 13:19:03.773912 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:03.773880    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-jxhsg" event={"ID":"0686fffb-a8a9-459f-878e-e434089eaf3d","Type":"ContainerStarted","Data":"c3f521173459cb0813896fc6b8eed652d65c7b29616f82724434331db2559e5a"}
Sep 08 13:19:03.776562 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:03.776532    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p84l2" event={"ID":"e84ecd52-fe2b-4159-99e1-dd97713d9a65","Type":"ContainerStarted","Data":"e16a5da6c473249fe6e5d4e6ccc6aaa9c042401a4e4ef444fcd02e929de4de84"}
Sep 08 13:19:03.776686 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:03.776610    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p84l2" event={"ID":"e84ecd52-fe2b-4159-99e1-dd97713d9a65","Type":"ContainerStarted","Data":"25b956f07a5b929414935b86e8ce410d3855401b40643a2d16be014175976724"}
Sep 08 13:19:03.800756 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:03.800700    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-p84l2" podStartSLOduration=2.551802448 podStartE2EDuration="7.800677726s" totalImagesPullingTime="5.178394216s" totalInitContainerRuntime="70.481062ms" isStatefulPod=true podCreationTimestamp="2026-09-08 13:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:03.798444728 +0000 UTC m=+207.250205656" watchObservedRunningTime="2026-09-08 13:19:03.800677726 +0000 UTC m=+207.252438658"
Sep 08 13:19:04.783584 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:04.783549    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h" event={"ID":"cc05686f-7483-4834-afcc-f89ce4721ccc","Type":"ContainerStarted","Data":"c11013c1e2650e34e1954743069cf4093736c408e4f8e89e684bcb7cc410e317"}
Sep 08 13:19:04.784040 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:04.783674    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h" event={"ID":"cc05686f-7483-4834-afcc-f89ce4721ccc","Type":"ContainerStarted","Data":"4d9a52933f039711acd3288fd8f3598551cc726c74f0ac215db3cb3cb3f4888d"}
Sep 08 13:19:04.784040 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:04.783710    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h" event={"ID":"cc05686f-7483-4834-afcc-f89ce4721ccc","Type":"ContainerStarted","Data":"bd185e72ea7aea12f9c0918c9e0588ce4e10e6d7e98978c707954f08586513c8"}
Sep 08 13:19:04.784040 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:04.783886    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h"
Sep 08 13:19:04.805929 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:04.805885    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h" podStartSLOduration=2.40047563 podStartE2EDuration="6.805867296s" totalImagesPullingTime="4.405391666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:04.804025681 +0000 UTC m=+208.255786625" watchObservedRunningTime="2026-09-08 13:19:04.805867296 +0000 UTC m=+208.257628226"
Sep 08 13:19:05.795704 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:05.795664    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ab57a63e-5ff7-4bdb-a5b9-70947b872230","Type":"ContainerStarted","Data":"210f204ecd35cbea3506ccc607d9a3eb58256fa28dfbf42352504f770406422b"}
Sep 08 13:19:05.796287 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:05.795751    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ab57a63e-5ff7-4bdb-a5b9-70947b872230","Type":"ContainerStarted","Data":"b1c1c7bba8b4df78087a6bcae8b163b3eb76cad72f28057096fe4e3dda370c4a"}
Sep 08 13:19:05.796287 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:05.795794    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ab57a63e-5ff7-4bdb-a5b9-70947b872230","Type":"ContainerStarted","Data":"8b5a026abb8d97c79f017ff178d93facc9383a962892e42858989a3406027159"}
Sep 08 13:19:05.796287 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:05.795835    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ab57a63e-5ff7-4bdb-a5b9-70947b872230","Type":"ContainerStarted","Data":"0fcfc6ee7b60b19687fdc83af2104bc45f18a9a6835c441832e2d62a66373ed9"}
Sep 08 13:19:05.796287 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:05.795877    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ab57a63e-5ff7-4bdb-a5b9-70947b872230","Type":"ContainerStarted","Data":"e45e82dc0cac4793ad71c368ca475ff20de861e95e2b09bd5e60b07e58a53d8a"}
Sep 08 13:19:05.796287 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:05.795919    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ab57a63e-5ff7-4bdb-a5b9-70947b872230","Type":"ContainerStarted","Data":"43436d9324522aed229d203ec06b668c1448732d5b96c9dea2f0202a626ff8dd"}
Sep 08 13:19:06.801392 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:06.801355    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-jxhsg" event={"ID":"0686fffb-a8a9-459f-878e-e434089eaf3d","Type":"ContainerStarted","Data":"97aa6aa006a0dc3e9b05e60ba5a959ac6b2a0bafa099606a840b415b3ae4f1c1"}
Sep 08 13:19:06.801829 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:06.801801    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-jxhsg"
Sep 08 13:19:06.806721 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:06.806695    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-jxhsg"
Sep 08 13:19:06.817662 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:06.817587    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-jxhsg" podStartSLOduration=2.912523716 podStartE2EDuration="5.8175653s" totalImagesPullingTime="2.905041584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:06.817401138 +0000 UTC m=+210.269162069" watchObservedRunningTime="2026-09-08 13:19:06.8175653 +0000 UTC m=+210.269326233"
Sep 08 13:19:06.818448 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:06.818406    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.070026955 podStartE2EDuration="9.818395327s" totalImagesPullingTime="4.729952735s" totalInitContainerRuntime="18.415637ms" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:05.830472017 +0000 UTC m=+209.282232962" watchObservedRunningTime="2026-09-08 13:19:06.818395327 +0000 UTC m=+210.270156269"
Sep 08 13:19:08.811668 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:08.811532    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-787d46f89f-pgss5" event={"ID":"f553b081-188a-47fa-9cac-a798cef2d48f","Type":"ContainerStarted","Data":"c471f57657196df358f14aab4e5ccc449996343cd611ea25f67c6607910c6076"}
Sep 08 13:19:08.811668 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:08.811618    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-787d46f89f-pgss5" event={"ID":"f553b081-188a-47fa-9cac-a798cef2d48f","Type":"ContainerStarted","Data":"253debda05aaec75f3af7a04487fcecbcac82b0e8e16056ab299e904f7778eb3"}
Sep 08 13:19:08.811668 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:08.811647    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-787d46f89f-pgss5" event={"ID":"f553b081-188a-47fa-9cac-a798cef2d48f","Type":"ContainerStarted","Data":"2556c368209d5ebdfa5519277546d0a20ebac51b861912403a7761f0d8dc6741"}
Sep 08 13:19:08.838149 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:08.838095    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-787d46f89f-pgss5" podStartSLOduration=1.897612632 podStartE2EDuration="7.838076803s" totalImagesPullingTime="5.940464171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:08.836388324 +0000 UTC m=+212.288149267" watchObservedRunningTime="2026-09-08 13:19:08.838076803 +0000 UTC m=+212.289837733"
Sep 08 13:19:08.975099 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:08.975067    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65c898f758-b2xjj"]
Sep 08 13:19:08.997924 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:08.997894    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65c898f758-b2xjj"]
Sep 08 13:19:08.998070 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:08.997990    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65c898f758-b2xjj"
Sep 08 13:19:09.006043 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:09.006017    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 08 13:19:09.087427 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:09.087342    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e47f6c2a-615f-4357-beca-dd4b75b03b21-console-oauth-config\") pod \"console-65c898f758-b2xjj\" (UID: \"e47f6c2a-615f-4357-beca-dd4b75b03b21\") " pod="openshift-console/console-65c898f758-b2xjj"
Sep 08 13:19:09.087427 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:09.087395    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e47f6c2a-615f-4357-beca-dd4b75b03b21-console-serving-cert\") pod \"console-65c898f758-b2xjj\" (UID: \"e47f6c2a-615f-4357-beca-dd4b75b03b21\") " pod="openshift-console/console-65c898f758-b2xjj"
Sep 08 13:19:09.087646 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:09.087435    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e47f6c2a-615f-4357-beca-dd4b75b03b21-oauth-serving-cert\") pod \"console-65c898f758-b2xjj\" (UID: \"e47f6c2a-615f-4357-beca-dd4b75b03b21\") " pod="openshift-console/console-65c898f758-b2xjj"
Sep 08 13:19:09.087646 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:09.087522    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e47f6c2a-615f-4357-beca-dd4b75b03b21-service-ca\") pod \"console-65c898f758-b2xjj\" (UID: \"e47f6c2a-615f-4357-beca-dd4b75b03b21\") " pod="openshift-console/console-65c898f758-b2xjj"
Sep 08 13:19:09.087646 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:09.087569    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e47f6c2a-615f-4357-beca-dd4b75b03b21-console-config\") pod \"console-65c898f758-b2xjj\" (UID: \"e47f6c2a-615f-4357-beca-dd4b75b03b21\") " pod="openshift-console/console-65c898f758-b2xjj"
Sep 08 13:19:09.087793 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:09.087645    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e47f6c2a-615f-4357-beca-dd4b75b03b21-trusted-ca-bundle\") pod \"console-65c898f758-b2xjj\" (UID: \"e47f6c2a-615f-4357-beca-dd4b75b03b21\") " pod="openshift-console/console-65c898f758-b2xjj"
Sep 08 13:19:09.087793 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:09.087686    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f2d5\" (UniqueName: \"kubernetes.io/projected/e47f6c2a-615f-4357-beca-dd4b75b03b21-kube-api-access-9f2d5\") pod \"console-65c898f758-b2xjj\" (UID: \"e47f6c2a-615f-4357-beca-dd4b75b03b21\") " pod="openshift-console/console-65c898f758-b2xjj"
Sep 08 13:19:09.188322 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:09.188288    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e47f6c2a-615f-4357-beca-dd4b75b03b21-console-oauth-config\") pod \"console-65c898f758-b2xjj\" (UID: \"e47f6c2a-615f-4357-beca-dd4b75b03b21\") " pod="openshift-console/console-65c898f758-b2xjj"
Sep 08 13:19:09.188482 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:09.188338    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e47f6c2a-615f-4357-beca-dd4b75b03b21-console-serving-cert\") pod \"console-65c898f758-b2xjj\" (UID: \"e47f6c2a-615f-4357-beca-dd4b75b03b21\") " pod="openshift-console/console-65c898f758-b2xjj"
Sep 08 13:19:09.188482 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:09.188370    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e47f6c2a-615f-4357-beca-dd4b75b03b21-oauth-serving-cert\") pod \"console-65c898f758-b2xjj\" (UID: \"e47f6c2a-615f-4357-beca-dd4b75b03b21\") " pod="openshift-console/console-65c898f758-b2xjj"
Sep 08 13:19:09.188482 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:09.188427    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e47f6c2a-615f-4357-beca-dd4b75b03b21-service-ca\") pod \"console-65c898f758-b2xjj\" (UID: \"e47f6c2a-615f-4357-beca-dd4b75b03b21\") " pod="openshift-console/console-65c898f758-b2xjj"
Sep 08 13:19:09.188482 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:09.188464    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e47f6c2a-615f-4357-beca-dd4b75b03b21-console-config\") pod \"console-65c898f758-b2xjj\" (UID: \"e47f6c2a-615f-4357-beca-dd4b75b03b21\") " pod="openshift-console/console-65c898f758-b2xjj"
Sep 08 13:19:09.188669 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:09.188492    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e47f6c2a-615f-4357-beca-dd4b75b03b21-trusted-ca-bundle\") pod \"console-65c898f758-b2xjj\" (UID: \"e47f6c2a-615f-4357-beca-dd4b75b03b21\") " pod="openshift-console/console-65c898f758-b2xjj"
Sep 08 13:19:09.188669 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:09.188520    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f2d5\" (UniqueName: \"kubernetes.io/projected/e47f6c2a-615f-4357-beca-dd4b75b03b21-kube-api-access-9f2d5\") pod \"console-65c898f758-b2xjj\" (UID: \"e47f6c2a-615f-4357-beca-dd4b75b03b21\") " pod="openshift-console/console-65c898f758-b2xjj"
Sep 08 13:19:09.189297 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:09.189270    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e47f6c2a-615f-4357-beca-dd4b75b03b21-oauth-serving-cert\") pod \"console-65c898f758-b2xjj\" (UID: \"e47f6c2a-615f-4357-beca-dd4b75b03b21\") " pod="openshift-console/console-65c898f758-b2xjj"
Sep 08 13:19:09.189384 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:09.189270    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e47f6c2a-615f-4357-beca-dd4b75b03b21-service-ca\") pod \"console-65c898f758-b2xjj\" (UID: \"e47f6c2a-615f-4357-beca-dd4b75b03b21\") " pod="openshift-console/console-65c898f758-b2xjj"
Sep 08 13:19:09.189384 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:09.189315    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e47f6c2a-615f-4357-beca-dd4b75b03b21-console-config\") pod \"console-65c898f758-b2xjj\" (UID: \"e47f6c2a-615f-4357-beca-dd4b75b03b21\") " pod="openshift-console/console-65c898f758-b2xjj"
Sep 08 13:19:09.189465 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:09.189447    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e47f6c2a-615f-4357-beca-dd4b75b03b21-trusted-ca-bundle\") pod \"console-65c898f758-b2xjj\" (UID: \"e47f6c2a-615f-4357-beca-dd4b75b03b21\") " pod="openshift-console/console-65c898f758-b2xjj"
Sep 08 13:19:09.191374 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:09.191353    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e47f6c2a-615f-4357-beca-dd4b75b03b21-console-serving-cert\") pod \"console-65c898f758-b2xjj\" (UID: \"e47f6c2a-615f-4357-beca-dd4b75b03b21\") " pod="openshift-console/console-65c898f758-b2xjj"
Sep 08 13:19:09.191374 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:09.191370    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e47f6c2a-615f-4357-beca-dd4b75b03b21-console-oauth-config\") pod \"console-65c898f758-b2xjj\" (UID: \"e47f6c2a-615f-4357-beca-dd4b75b03b21\") " pod="openshift-console/console-65c898f758-b2xjj"
Sep 08 13:19:09.196515 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:09.196493    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f2d5\" (UniqueName: \"kubernetes.io/projected/e47f6c2a-615f-4357-beca-dd4b75b03b21-kube-api-access-9f2d5\") pod \"console-65c898f758-b2xjj\" (UID: \"e47f6c2a-615f-4357-beca-dd4b75b03b21\") " pod="openshift-console/console-65c898f758-b2xjj"
Sep 08 13:19:09.307636 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:09.307576    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65c898f758-b2xjj"
Sep 08 13:19:09.615153 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:09.615125    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65c898f758-b2xjj"]
Sep 08 13:19:09.617692 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:19:09.617665    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode47f6c2a_615f_4357_beca_dd4b75b03b21.slice/crio-f5844fb878905ba1bd55afbd3dcb8b628349954934d71a6c7dfe13db677d7f06 WatchSource:0}: Error finding container f5844fb878905ba1bd55afbd3dcb8b628349954934d71a6c7dfe13db677d7f06: Status 404 returned error can't find the container with id f5844fb878905ba1bd55afbd3dcb8b628349954934d71a6c7dfe13db677d7f06
Sep 08 13:19:09.818363 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:09.818321    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65c898f758-b2xjj" event={"ID":"e47f6c2a-615f-4357-beca-dd4b75b03b21","Type":"ContainerStarted","Data":"2fc5f6f545e6e3ca0e3c62b5b2d8121fc48e876f8fbb8788acdf2cbdb435eaff"}
Sep 08 13:19:09.818830 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:09.818380    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65c898f758-b2xjj" event={"ID":"e47f6c2a-615f-4357-beca-dd4b75b03b21","Type":"ContainerStarted","Data":"f5844fb878905ba1bd55afbd3dcb8b628349954934d71a6c7dfe13db677d7f06"}
Sep 08 13:19:09.844319 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:09.844263    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65c898f758-b2xjj" podStartSLOduration=1.844245176 podStartE2EDuration="1.844245176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:09.842859351 +0000 UTC m=+213.294620303" watchObservedRunningTime="2026-09-08 13:19:09.844245176 +0000 UTC m=+213.296006108"
Sep 08 13:19:10.802244 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:10.802211    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-cd4f7799f-bnz9h"
Sep 08 13:19:16.550056 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:16.549992    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7" podUID="494d9fb4-e6b1-4d95-a27d-194c4d4c6079" containerName="registry" containerID="cri-o://7be584a87cefbed42a75e29ce4e097d62b9447199a7b8d32d0887a67c19920b9" gracePeriod=30
Sep 08 13:19:16.803280 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:16.803212    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:19:16.845452 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:16.845417    2558 generic.go:356] "Generic (PLEG): container finished" podID="494d9fb4-e6b1-4d95-a27d-194c4d4c6079" containerID="7be584a87cefbed42a75e29ce4e097d62b9447199a7b8d32d0887a67c19920b9" exitCode=0
Sep 08 13:19:16.845658 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:16.845458    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7"
Sep 08 13:19:16.845658 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:16.845503    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7" event={"ID":"494d9fb4-e6b1-4d95-a27d-194c4d4c6079","Type":"ContainerDied","Data":"7be584a87cefbed42a75e29ce4e097d62b9447199a7b8d32d0887a67c19920b9"}
Sep 08 13:19:16.845658 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:16.845556    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-bcb89cbc9-g9vr7" event={"ID":"494d9fb4-e6b1-4d95-a27d-194c4d4c6079","Type":"ContainerDied","Data":"012d1e400e3508b88e02cbe424f7c391bf56aba1a9a1fdd60f36ad4a80ea08e7"}
Sep 08 13:19:16.845658 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:16.845577    2558 scope.go:117] "RemoveContainer" containerID="7be584a87cefbed42a75e29ce4e097d62b9447199a7b8d32d0887a67c19920b9"
Sep 08 13:19:16.853685 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:16.853664    2558 scope.go:117] "RemoveContainer" containerID="7be584a87cefbed42a75e29ce4e097d62b9447199a7b8d32d0887a67c19920b9"
Sep 08 13:19:16.853995 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:19:16.853968    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7be584a87cefbed42a75e29ce4e097d62b9447199a7b8d32d0887a67c19920b9\": container with ID starting with 7be584a87cefbed42a75e29ce4e097d62b9447199a7b8d32d0887a67c19920b9 not found: ID does not exist" containerID="7be584a87cefbed42a75e29ce4e097d62b9447199a7b8d32d0887a67c19920b9"
Sep 08 13:19:16.854065 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:16.854003    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7be584a87cefbed42a75e29ce4e097d62b9447199a7b8d32d0887a67c19920b9"} err="failed to get container status \"7be584a87cefbed42a75e29ce4e097d62b9447199a7b8d32d0887a67c19920b9\": rpc error: code = NotFound desc = could not find container \"7be584a87cefbed42a75e29ce4e097d62b9447199a7b8d32d0887a67c19920b9\": container with ID starting with 7be584a87cefbed42a75e29ce4e097d62b9447199a7b8d32d0887a67c19920b9 not found: ID does not exist"
Sep 08 13:19:16.856281 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:16.856260    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-tls\") pod \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") "
Sep 08 13:19:16.856342 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:16.856298    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frtr6\" (UniqueName: \"kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-kube-api-access-frtr6\") pod \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") "
Sep 08 13:19:16.856342 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:16.856321    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-image-registry-private-configuration\") pod \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") "
Sep 08 13:19:16.856409 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:16.856350    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-certificates\") pod \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") "
Sep 08 13:19:16.856448 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:16.856404    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-ca-trust-extracted\") pod \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") "
Sep 08 13:19:16.856487 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:16.856467    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-trusted-ca\") pod \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") "
Sep 08 13:19:16.856540 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:16.856492    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-bound-sa-token\") pod \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") "
Sep 08 13:19:16.856605 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:16.856542    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-installation-pull-secrets\") pod \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\" (UID: \"494d9fb4-e6b1-4d95-a27d-194c4d4c6079\") "
Sep 08 13:19:16.856861 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:16.856821    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "494d9fb4-e6b1-4d95-a27d-194c4d4c6079" (UID: "494d9fb4-e6b1-4d95-a27d-194c4d4c6079"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:19:16.856978 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:16.856885    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "494d9fb4-e6b1-4d95-a27d-194c4d4c6079" (UID: "494d9fb4-e6b1-4d95-a27d-194c4d4c6079"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:19:16.858884 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:16.858854    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-kube-api-access-frtr6" (OuterVolumeSpecName: "kube-api-access-frtr6") pod "494d9fb4-e6b1-4d95-a27d-194c4d4c6079" (UID: "494d9fb4-e6b1-4d95-a27d-194c4d4c6079"). InnerVolumeSpecName "kube-api-access-frtr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:19:16.859001 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:16.858978    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "494d9fb4-e6b1-4d95-a27d-194c4d4c6079" (UID: "494d9fb4-e6b1-4d95-a27d-194c4d4c6079"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:19:16.859066 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:16.859001    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "494d9fb4-e6b1-4d95-a27d-194c4d4c6079" (UID: "494d9fb4-e6b1-4d95-a27d-194c4d4c6079"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:19:16.859186 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:16.859159    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "494d9fb4-e6b1-4d95-a27d-194c4d4c6079" (UID: "494d9fb4-e6b1-4d95-a27d-194c4d4c6079"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:19:16.859290 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:16.859268    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "494d9fb4-e6b1-4d95-a27d-194c4d4c6079" (UID: "494d9fb4-e6b1-4d95-a27d-194c4d4c6079"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:19:16.865216 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:16.865186    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "494d9fb4-e6b1-4d95-a27d-194c4d4c6079" (UID: "494d9fb4-e6b1-4d95-a27d-194c4d4c6079"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:19:16.957273 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:16.957238    2558 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-installation-pull-secrets\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:19:16.957273 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:16.957269    2558 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-tls\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:19:16.957273 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:16.957278    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frtr6\" (UniqueName: \"kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-kube-api-access-frtr6\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:19:16.957512 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:16.957288    2558 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-image-registry-private-configuration\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:19:16.957512 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:16.957296    2558 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-registry-certificates\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:19:16.957512 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:16.957305    2558 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-ca-trust-extracted\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:19:16.957512 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:16.957313    2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-trusted-ca\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:19:16.957512 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:16.957321    2558 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/494d9fb4-e6b1-4d95-a27d-194c4d4c6079-bound-sa-token\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:19:17.173274 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:17.173229    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-bcb89cbc9-g9vr7"]
Sep 08 13:19:17.179317 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:17.179288    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-bcb89cbc9-g9vr7"]
Sep 08 13:19:19.052846 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:19.052803    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="494d9fb4-e6b1-4d95-a27d-194c4d4c6079" path="/var/lib/kubelet/pods/494d9fb4-e6b1-4d95-a27d-194c4d4c6079/volumes"
Sep 08 13:19:19.308057 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:19.307976    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65c898f758-b2xjj"
Sep 08 13:19:19.308057 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:19.308031    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65c898f758-b2xjj"
Sep 08 13:19:19.312784 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:19.312763    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65c898f758-b2xjj"
Sep 08 13:19:19.861830 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:19.861803    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65c898f758-b2xjj"
Sep 08 13:19:20.650044 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:20.649980    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-576bd68b6b-qpsch" podUID="eac540e7-77f1-49ba-b3da-b875082bcaea" containerName="console" containerID="cri-o://028522ec7914ccdcc004320d353dea0425fd7c2c36789e3da8993dd043cb125c" gracePeriod=15
Sep 08 13:19:20.863779 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:20.863741    2558 generic.go:356] "Generic (PLEG): container finished" podID="eac540e7-77f1-49ba-b3da-b875082bcaea" containerID="028522ec7914ccdcc004320d353dea0425fd7c2c36789e3da8993dd043cb125c" exitCode=0
Sep 08 13:19:20.863926 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:20.863810    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-576bd68b6b-qpsch" event={"ID":"eac540e7-77f1-49ba-b3da-b875082bcaea","Type":"ContainerDied","Data":"028522ec7914ccdcc004320d353dea0425fd7c2c36789e3da8993dd043cb125c"}
Sep 08 13:19:20.930359 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:20.930280    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-576bd68b6b-qpsch"
Sep 08 13:19:20.991359 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:20.991328    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwcqk\" (UniqueName: \"kubernetes.io/projected/eac540e7-77f1-49ba-b3da-b875082bcaea-kube-api-access-fwcqk\") pod \"eac540e7-77f1-49ba-b3da-b875082bcaea\" (UID: \"eac540e7-77f1-49ba-b3da-b875082bcaea\") "
Sep 08 13:19:20.991359 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:20.991370    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eac540e7-77f1-49ba-b3da-b875082bcaea-console-oauth-config\") pod \"eac540e7-77f1-49ba-b3da-b875082bcaea\" (UID: \"eac540e7-77f1-49ba-b3da-b875082bcaea\") "
Sep 08 13:19:20.991570 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:20.991388    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eac540e7-77f1-49ba-b3da-b875082bcaea-console-config\") pod \"eac540e7-77f1-49ba-b3da-b875082bcaea\" (UID: \"eac540e7-77f1-49ba-b3da-b875082bcaea\") "
Sep 08 13:19:20.991570 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:20.991457    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eac540e7-77f1-49ba-b3da-b875082bcaea-service-ca\") pod \"eac540e7-77f1-49ba-b3da-b875082bcaea\" (UID: \"eac540e7-77f1-49ba-b3da-b875082bcaea\") "
Sep 08 13:19:20.991570 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:20.991486    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eac540e7-77f1-49ba-b3da-b875082bcaea-console-serving-cert\") pod \"eac540e7-77f1-49ba-b3da-b875082bcaea\" (UID: \"eac540e7-77f1-49ba-b3da-b875082bcaea\") "
Sep 08 13:19:20.991570 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:20.991508    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eac540e7-77f1-49ba-b3da-b875082bcaea-oauth-serving-cert\") pod \"eac540e7-77f1-49ba-b3da-b875082bcaea\" (UID: \"eac540e7-77f1-49ba-b3da-b875082bcaea\") "
Sep 08 13:19:20.991890 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:20.991861    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eac540e7-77f1-49ba-b3da-b875082bcaea-console-config" (OuterVolumeSpecName: "console-config") pod "eac540e7-77f1-49ba-b3da-b875082bcaea" (UID: "eac540e7-77f1-49ba-b3da-b875082bcaea"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:19:20.992017 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:20.991956    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eac540e7-77f1-49ba-b3da-b875082bcaea-service-ca" (OuterVolumeSpecName: "service-ca") pod "eac540e7-77f1-49ba-b3da-b875082bcaea" (UID: "eac540e7-77f1-49ba-b3da-b875082bcaea"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:19:20.992017 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:20.991960    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eac540e7-77f1-49ba-b3da-b875082bcaea-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "eac540e7-77f1-49ba-b3da-b875082bcaea" (UID: "eac540e7-77f1-49ba-b3da-b875082bcaea"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:19:20.993839 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:20.993802    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eac540e7-77f1-49ba-b3da-b875082bcaea-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "eac540e7-77f1-49ba-b3da-b875082bcaea" (UID: "eac540e7-77f1-49ba-b3da-b875082bcaea"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:19:20.993839 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:20.993807    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eac540e7-77f1-49ba-b3da-b875082bcaea-kube-api-access-fwcqk" (OuterVolumeSpecName: "kube-api-access-fwcqk") pod "eac540e7-77f1-49ba-b3da-b875082bcaea" (UID: "eac540e7-77f1-49ba-b3da-b875082bcaea"). InnerVolumeSpecName "kube-api-access-fwcqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:19:20.993977 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:20.993875    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eac540e7-77f1-49ba-b3da-b875082bcaea-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "eac540e7-77f1-49ba-b3da-b875082bcaea" (UID: "eac540e7-77f1-49ba-b3da-b875082bcaea"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:19:21.092370 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:21.092329    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwcqk\" (UniqueName: \"kubernetes.io/projected/eac540e7-77f1-49ba-b3da-b875082bcaea-kube-api-access-fwcqk\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:19:21.092370 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:21.092360    2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eac540e7-77f1-49ba-b3da-b875082bcaea-console-oauth-config\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:19:21.092370 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:21.092371    2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eac540e7-77f1-49ba-b3da-b875082bcaea-console-config\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:19:21.092370 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:21.092380    2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eac540e7-77f1-49ba-b3da-b875082bcaea-service-ca\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:19:21.092644 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:21.092388    2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eac540e7-77f1-49ba-b3da-b875082bcaea-console-serving-cert\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:19:21.092644 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:21.092396    2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eac540e7-77f1-49ba-b3da-b875082bcaea-oauth-serving-cert\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:19:21.869105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:21.869077    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-576bd68b6b-qpsch"
Sep 08 13:19:21.869487 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:21.869069    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-576bd68b6b-qpsch" event={"ID":"eac540e7-77f1-49ba-b3da-b875082bcaea","Type":"ContainerDied","Data":"bf51a99a9b4a74306a80107a1e2ae21ea68eafcbd4125330cc4bebb43706db77"}
Sep 08 13:19:21.869487 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:21.869208    2558 scope.go:117] "RemoveContainer" containerID="028522ec7914ccdcc004320d353dea0425fd7c2c36789e3da8993dd043cb125c"
Sep 08 13:19:21.890059 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:21.890032    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-576bd68b6b-qpsch"]
Sep 08 13:19:21.893662 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:21.893636    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-576bd68b6b-qpsch"]
Sep 08 13:19:23.052695 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:23.052663    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eac540e7-77f1-49ba-b3da-b875082bcaea" path="/var/lib/kubelet/pods/eac540e7-77f1-49ba-b3da-b875082bcaea/volumes"
Sep 08 13:19:29.902282 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:29.902248    2558 generic.go:356] "Generic (PLEG): container finished" podID="e818a776-d654-41ac-ba74-e47a0532f4d6" containerID="0c66008a3b78ff5c44a7a94f49775aa94580d16f2a8783f99f2b4b11935f2ff0" exitCode=0
Sep 08 13:19:29.902713 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:29.902323    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-4q6fg" event={"ID":"e818a776-d654-41ac-ba74-e47a0532f4d6","Type":"ContainerDied","Data":"0c66008a3b78ff5c44a7a94f49775aa94580d16f2a8783f99f2b4b11935f2ff0"}
Sep 08 13:19:29.902713 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:29.902667    2558 scope.go:117] "RemoveContainer" containerID="0c66008a3b78ff5c44a7a94f49775aa94580d16f2a8783f99f2b4b11935f2ff0"
Sep 08 13:19:30.907777 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:30.907740    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-4q6fg" event={"ID":"e818a776-d654-41ac-ba74-e47a0532f4d6","Type":"ContainerStarted","Data":"4bf9981068e3ee9108e49a41034b3abe329e6f1302e710e726fb0cf4fa8fe5b1"}
Sep 08 13:19:48.839653 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:48.839605    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90dadd3f-c403-4d64-af3a-45b4c68163e5-metrics-certs\") pod \"network-metrics-daemon-q4m28\" (UID: \"90dadd3f-c403-4d64-af3a-45b4c68163e5\") " pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:19:48.841950 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:48.841929    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90dadd3f-c403-4d64-af3a-45b4c68163e5-metrics-certs\") pod \"network-metrics-daemon-q4m28\" (UID: \"90dadd3f-c403-4d64-af3a-45b4c68163e5\") " pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:19:49.051375 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:49.051344    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4rwvw\""
Sep 08 13:19:49.059432 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:49.059409    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4m28"
Sep 08 13:19:49.334091 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:49.334059    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q4m28"]
Sep 08 13:19:49.338224 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:19:49.338196    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90dadd3f_c403_4d64_af3a_45b4c68163e5.slice/crio-0f3b022d4e8c28ed891347b7357f0dea07a9716058a6875f5014aa30761613ff WatchSource:0}: Error finding container 0f3b022d4e8c28ed891347b7357f0dea07a9716058a6875f5014aa30761613ff: Status 404 returned error can't find the container with id 0f3b022d4e8c28ed891347b7357f0dea07a9716058a6875f5014aa30761613ff
Sep 08 13:19:49.985690 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:49.985652    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q4m28" event={"ID":"90dadd3f-c403-4d64-af3a-45b4c68163e5","Type":"ContainerStarted","Data":"0f3b022d4e8c28ed891347b7357f0dea07a9716058a6875f5014aa30761613ff"}
Sep 08 13:19:50.991703 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:50.991666    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q4m28" event={"ID":"90dadd3f-c403-4d64-af3a-45b4c68163e5","Type":"ContainerStarted","Data":"e1bd4e9a822ec507a75d3b295ee1bab33d72d11a7a7e846a7c70c354f794d560"}
Sep 08 13:19:50.992117 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:50.991727    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q4m28" event={"ID":"90dadd3f-c403-4d64-af3a-45b4c68163e5","Type":"ContainerStarted","Data":"a84be969605e6ff93b83da5da6f14b4ab4178817573bfbca6b6c1c9b3b2fc3b4"}
Sep 08 13:19:51.009321 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:19:51.009275    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-q4m28" podStartSLOduration=252.565157839 podStartE2EDuration="4m14.009259355s" totalImagesPullingTime="1.444101516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:15:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:51.007388631 +0000 UTC m=+254.459149565" watchObservedRunningTime="2026-09-08 13:19:51.009259355 +0000 UTC m=+254.461020287"
Sep 08 13:20:13.008227 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:13.008127    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c9d476b6c-ksvld"]
Sep 08 13:20:13.008678 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:13.008430    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eac540e7-77f1-49ba-b3da-b875082bcaea" containerName="console"
Sep 08 13:20:13.008678 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:13.008440    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="eac540e7-77f1-49ba-b3da-b875082bcaea" containerName="console"
Sep 08 13:20:13.008678 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:13.008458    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="494d9fb4-e6b1-4d95-a27d-194c4d4c6079" containerName="registry"
Sep 08 13:20:13.008678 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:13.008464    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="494d9fb4-e6b1-4d95-a27d-194c4d4c6079" containerName="registry"
Sep 08 13:20:13.008678 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:13.008531    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="494d9fb4-e6b1-4d95-a27d-194c4d4c6079" containerName="registry"
Sep 08 13:20:13.008678 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:13.008538    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="eac540e7-77f1-49ba-b3da-b875082bcaea" containerName="console"
Sep 08 13:20:13.012841 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:13.012816    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9d476b6c-ksvld"
Sep 08 13:20:13.032312 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:13.032281    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c9d476b6c-ksvld"]
Sep 08 13:20:13.112112 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:13.112076    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/276be51b-30a3-468b-be17-7a67a4654752-oauth-serving-cert\") pod \"console-7c9d476b6c-ksvld\" (UID: \"276be51b-30a3-468b-be17-7a67a4654752\") " pod="openshift-console/console-7c9d476b6c-ksvld"
Sep 08 13:20:13.112112 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:13.112113    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/276be51b-30a3-468b-be17-7a67a4654752-service-ca\") pod \"console-7c9d476b6c-ksvld\" (UID: \"276be51b-30a3-468b-be17-7a67a4654752\") " pod="openshift-console/console-7c9d476b6c-ksvld"
Sep 08 13:20:13.112306 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:13.112134    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/276be51b-30a3-468b-be17-7a67a4654752-console-serving-cert\") pod \"console-7c9d476b6c-ksvld\" (UID: \"276be51b-30a3-468b-be17-7a67a4654752\") " pod="openshift-console/console-7c9d476b6c-ksvld"
Sep 08 13:20:13.112306 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:13.112205    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/276be51b-30a3-468b-be17-7a67a4654752-console-config\") pod \"console-7c9d476b6c-ksvld\" (UID: \"276be51b-30a3-468b-be17-7a67a4654752\") " pod="openshift-console/console-7c9d476b6c-ksvld"
Sep 08 13:20:13.112306 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:13.112229    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/276be51b-30a3-468b-be17-7a67a4654752-console-oauth-config\") pod \"console-7c9d476b6c-ksvld\" (UID: \"276be51b-30a3-468b-be17-7a67a4654752\") " pod="openshift-console/console-7c9d476b6c-ksvld"
Sep 08 13:20:13.112306 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:13.112283    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/276be51b-30a3-468b-be17-7a67a4654752-trusted-ca-bundle\") pod \"console-7c9d476b6c-ksvld\" (UID: \"276be51b-30a3-468b-be17-7a67a4654752\") " pod="openshift-console/console-7c9d476b6c-ksvld"
Sep 08 13:20:13.112437 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:13.112335    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzbt6\" (UniqueName: \"kubernetes.io/projected/276be51b-30a3-468b-be17-7a67a4654752-kube-api-access-jzbt6\") pod \"console-7c9d476b6c-ksvld\" (UID: \"276be51b-30a3-468b-be17-7a67a4654752\") " pod="openshift-console/console-7c9d476b6c-ksvld"
Sep 08 13:20:13.213022 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:13.212984    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/276be51b-30a3-468b-be17-7a67a4654752-console-config\") pod \"console-7c9d476b6c-ksvld\" (UID: \"276be51b-30a3-468b-be17-7a67a4654752\") " pod="openshift-console/console-7c9d476b6c-ksvld"
Sep 08 13:20:13.213022 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:13.213028    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/276be51b-30a3-468b-be17-7a67a4654752-console-oauth-config\") pod \"console-7c9d476b6c-ksvld\" (UID: \"276be51b-30a3-468b-be17-7a67a4654752\") " pod="openshift-console/console-7c9d476b6c-ksvld"
Sep 08 13:20:13.213286 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:13.213055    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/276be51b-30a3-468b-be17-7a67a4654752-trusted-ca-bundle\") pod \"console-7c9d476b6c-ksvld\" (UID: \"276be51b-30a3-468b-be17-7a67a4654752\") " pod="openshift-console/console-7c9d476b6c-ksvld"
Sep 08 13:20:13.213286 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:13.213091    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzbt6\" (UniqueName: \"kubernetes.io/projected/276be51b-30a3-468b-be17-7a67a4654752-kube-api-access-jzbt6\") pod \"console-7c9d476b6c-ksvld\" (UID: \"276be51b-30a3-468b-be17-7a67a4654752\") " pod="openshift-console/console-7c9d476b6c-ksvld"
Sep 08 13:20:13.213286 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:13.213118    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/276be51b-30a3-468b-be17-7a67a4654752-oauth-serving-cert\") pod \"console-7c9d476b6c-ksvld\" (UID: \"276be51b-30a3-468b-be17-7a67a4654752\") " pod="openshift-console/console-7c9d476b6c-ksvld"
Sep 08 13:20:13.213286 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:13.213143    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/276be51b-30a3-468b-be17-7a67a4654752-service-ca\") pod \"console-7c9d476b6c-ksvld\" (UID: \"276be51b-30a3-468b-be17-7a67a4654752\") " pod="openshift-console/console-7c9d476b6c-ksvld"
Sep 08 13:20:13.213286 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:13.213172    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/276be51b-30a3-468b-be17-7a67a4654752-console-serving-cert\") pod \"console-7c9d476b6c-ksvld\" (UID: \"276be51b-30a3-468b-be17-7a67a4654752\") " pod="openshift-console/console-7c9d476b6c-ksvld"
Sep 08 13:20:13.214035 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:13.213998    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/276be51b-30a3-468b-be17-7a67a4654752-oauth-serving-cert\") pod \"console-7c9d476b6c-ksvld\" (UID: \"276be51b-30a3-468b-be17-7a67a4654752\") " pod="openshift-console/console-7c9d476b6c-ksvld"
Sep 08 13:20:13.214171 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:13.214100    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/276be51b-30a3-468b-be17-7a67a4654752-trusted-ca-bundle\") pod \"console-7c9d476b6c-ksvld\" (UID: \"276be51b-30a3-468b-be17-7a67a4654752\") " pod="openshift-console/console-7c9d476b6c-ksvld"
Sep 08 13:20:13.214171 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:13.214155    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/276be51b-30a3-468b-be17-7a67a4654752-service-ca\") pod \"console-7c9d476b6c-ksvld\" (UID: \"276be51b-30a3-468b-be17-7a67a4654752\") " pod="openshift-console/console-7c9d476b6c-ksvld"
Sep 08 13:20:13.214375 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:13.214356    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/276be51b-30a3-468b-be17-7a67a4654752-console-config\") pod \"console-7c9d476b6c-ksvld\" (UID: \"276be51b-30a3-468b-be17-7a67a4654752\") " pod="openshift-console/console-7c9d476b6c-ksvld"
Sep 08 13:20:13.215813 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:13.215794    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/276be51b-30a3-468b-be17-7a67a4654752-console-serving-cert\") pod \"console-7c9d476b6c-ksvld\" (UID: \"276be51b-30a3-468b-be17-7a67a4654752\") " pod="openshift-console/console-7c9d476b6c-ksvld"
Sep 08 13:20:13.216000 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:13.215980    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/276be51b-30a3-468b-be17-7a67a4654752-console-oauth-config\") pod \"console-7c9d476b6c-ksvld\" (UID: \"276be51b-30a3-468b-be17-7a67a4654752\") " pod="openshift-console/console-7c9d476b6c-ksvld"
Sep 08 13:20:13.221351 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:13.221321    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzbt6\" (UniqueName: \"kubernetes.io/projected/276be51b-30a3-468b-be17-7a67a4654752-kube-api-access-jzbt6\") pod \"console-7c9d476b6c-ksvld\" (UID: \"276be51b-30a3-468b-be17-7a67a4654752\") " pod="openshift-console/console-7c9d476b6c-ksvld"
Sep 08 13:20:13.322847 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:13.322740    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9d476b6c-ksvld"
Sep 08 13:20:13.598679 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:13.598572    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c9d476b6c-ksvld"]
Sep 08 13:20:13.601669 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:20:13.601634    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod276be51b_30a3_468b_be17_7a67a4654752.slice/crio-2c2b6a8c7f895644b657cfe8844e1a80a55b8b9a85bbd76b0072afb7530dfe32 WatchSource:0}: Error finding container 2c2b6a8c7f895644b657cfe8844e1a80a55b8b9a85bbd76b0072afb7530dfe32: Status 404 returned error can't find the container with id 2c2b6a8c7f895644b657cfe8844e1a80a55b8b9a85bbd76b0072afb7530dfe32
Sep 08 13:20:14.087998 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:14.087966    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9d476b6c-ksvld" event={"ID":"276be51b-30a3-468b-be17-7a67a4654752","Type":"ContainerStarted","Data":"c8c770a4998d6813e25abadee7692dccc9963be2e14d79139565a67318465339"}
Sep 08 13:20:14.088390 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:14.088022    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9d476b6c-ksvld" event={"ID":"276be51b-30a3-468b-be17-7a67a4654752","Type":"ContainerStarted","Data":"2c2b6a8c7f895644b657cfe8844e1a80a55b8b9a85bbd76b0072afb7530dfe32"}
Sep 08 13:20:14.109263 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:14.109208    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c9d476b6c-ksvld" podStartSLOduration=2.109190184 podStartE2EDuration="2.109190184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:20:14.107860346 +0000 UTC m=+277.559621279" watchObservedRunningTime="2026-09-08 13:20:14.109190184 +0000 UTC m=+277.560951114"
Sep 08 13:20:23.323296 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:23.323254    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c9d476b6c-ksvld"
Sep 08 13:20:23.323856 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:23.323363    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c9d476b6c-ksvld"
Sep 08 13:20:23.327942 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:23.327915    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c9d476b6c-ksvld"
Sep 08 13:20:24.130616 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:24.130573    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c9d476b6c-ksvld"
Sep 08 13:20:24.184643 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:24.184605    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65c898f758-b2xjj"]
Sep 08 13:20:37.005523 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:37.005494    2558 kubelet.go:1648] "Image garbage collection succeeded"
Sep 08 13:20:49.204783 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:49.204735    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-65c898f758-b2xjj" podUID="e47f6c2a-615f-4357-beca-dd4b75b03b21" containerName="console" containerID="cri-o://2fc5f6f545e6e3ca0e3c62b5b2d8121fc48e876f8fbb8788acdf2cbdb435eaff" gracePeriod=15
Sep 08 13:20:49.455352 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:49.455295    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65c898f758-b2xjj"
Sep 08 13:20:49.617655 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:49.617622    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e47f6c2a-615f-4357-beca-dd4b75b03b21-console-config\") pod \"e47f6c2a-615f-4357-beca-dd4b75b03b21\" (UID: \"e47f6c2a-615f-4357-beca-dd4b75b03b21\") "
Sep 08 13:20:49.617655 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:49.617658    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e47f6c2a-615f-4357-beca-dd4b75b03b21-oauth-serving-cert\") pod \"e47f6c2a-615f-4357-beca-dd4b75b03b21\" (UID: \"e47f6c2a-615f-4357-beca-dd4b75b03b21\") "
Sep 08 13:20:49.617856 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:49.617726    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e47f6c2a-615f-4357-beca-dd4b75b03b21-console-serving-cert\") pod \"e47f6c2a-615f-4357-beca-dd4b75b03b21\" (UID: \"e47f6c2a-615f-4357-beca-dd4b75b03b21\") "
Sep 08 13:20:49.617856 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:49.617747    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9f2d5\" (UniqueName: \"kubernetes.io/projected/e47f6c2a-615f-4357-beca-dd4b75b03b21-kube-api-access-9f2d5\") pod \"e47f6c2a-615f-4357-beca-dd4b75b03b21\" (UID: \"e47f6c2a-615f-4357-beca-dd4b75b03b21\") "
Sep 08 13:20:49.617856 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:49.617785    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e47f6c2a-615f-4357-beca-dd4b75b03b21-trusted-ca-bundle\") pod \"e47f6c2a-615f-4357-beca-dd4b75b03b21\" (UID: \"e47f6c2a-615f-4357-beca-dd4b75b03b21\") "
Sep 08 13:20:49.617856 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:49.617808    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e47f6c2a-615f-4357-beca-dd4b75b03b21-console-oauth-config\") pod \"e47f6c2a-615f-4357-beca-dd4b75b03b21\" (UID: \"e47f6c2a-615f-4357-beca-dd4b75b03b21\") "
Sep 08 13:20:49.617856 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:49.617846    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e47f6c2a-615f-4357-beca-dd4b75b03b21-service-ca\") pod \"e47f6c2a-615f-4357-beca-dd4b75b03b21\" (UID: \"e47f6c2a-615f-4357-beca-dd4b75b03b21\") "
Sep 08 13:20:49.618138 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:49.618049    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e47f6c2a-615f-4357-beca-dd4b75b03b21-console-config" (OuterVolumeSpecName: "console-config") pod "e47f6c2a-615f-4357-beca-dd4b75b03b21" (UID: "e47f6c2a-615f-4357-beca-dd4b75b03b21"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:20:49.618265 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:49.618195    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e47f6c2a-615f-4357-beca-dd4b75b03b21-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e47f6c2a-615f-4357-beca-dd4b75b03b21" (UID: "e47f6c2a-615f-4357-beca-dd4b75b03b21"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:20:49.618335 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:49.618259    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e47f6c2a-615f-4357-beca-dd4b75b03b21-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e47f6c2a-615f-4357-beca-dd4b75b03b21" (UID: "e47f6c2a-615f-4357-beca-dd4b75b03b21"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:20:49.618335 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:49.618295    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e47f6c2a-615f-4357-beca-dd4b75b03b21-service-ca" (OuterVolumeSpecName: "service-ca") pod "e47f6c2a-615f-4357-beca-dd4b75b03b21" (UID: "e47f6c2a-615f-4357-beca-dd4b75b03b21"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:20:49.619866 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:49.619843    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e47f6c2a-615f-4357-beca-dd4b75b03b21-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e47f6c2a-615f-4357-beca-dd4b75b03b21" (UID: "e47f6c2a-615f-4357-beca-dd4b75b03b21"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:20:49.619986 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:49.619966    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e47f6c2a-615f-4357-beca-dd4b75b03b21-kube-api-access-9f2d5" (OuterVolumeSpecName: "kube-api-access-9f2d5") pod "e47f6c2a-615f-4357-beca-dd4b75b03b21" (UID: "e47f6c2a-615f-4357-beca-dd4b75b03b21"). InnerVolumeSpecName "kube-api-access-9f2d5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:20:49.620038 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:49.620016    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e47f6c2a-615f-4357-beca-dd4b75b03b21-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e47f6c2a-615f-4357-beca-dd4b75b03b21" (UID: "e47f6c2a-615f-4357-beca-dd4b75b03b21"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:20:49.718920 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:49.718832    2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e47f6c2a-615f-4357-beca-dd4b75b03b21-trusted-ca-bundle\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:20:49.718920 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:49.718862    2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e47f6c2a-615f-4357-beca-dd4b75b03b21-console-oauth-config\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:20:49.718920 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:49.718875    2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e47f6c2a-615f-4357-beca-dd4b75b03b21-service-ca\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:20:49.718920 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:49.718885    2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e47f6c2a-615f-4357-beca-dd4b75b03b21-console-config\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:20:49.718920 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:49.718894    2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e47f6c2a-615f-4357-beca-dd4b75b03b21-oauth-serving-cert\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:20:49.718920 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:49.718903    2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e47f6c2a-615f-4357-beca-dd4b75b03b21-console-serving-cert\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:20:49.718920 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:49.718911    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f2d5\" (UniqueName: \"kubernetes.io/projected/e47f6c2a-615f-4357-beca-dd4b75b03b21-kube-api-access-9f2d5\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:20:50.239917 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:50.239879    2558 generic.go:356] "Generic (PLEG): container finished" podID="e47f6c2a-615f-4357-beca-dd4b75b03b21" containerID="2fc5f6f545e6e3ca0e3c62b5b2d8121fc48e876f8fbb8788acdf2cbdb435eaff" exitCode=0
Sep 08 13:20:50.240310 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:50.239924    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65c898f758-b2xjj"
Sep 08 13:20:50.240310 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:50.239967    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65c898f758-b2xjj" event={"ID":"e47f6c2a-615f-4357-beca-dd4b75b03b21","Type":"ContainerDied","Data":"2fc5f6f545e6e3ca0e3c62b5b2d8121fc48e876f8fbb8788acdf2cbdb435eaff"}
Sep 08 13:20:50.240310 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:50.240017    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65c898f758-b2xjj" event={"ID":"e47f6c2a-615f-4357-beca-dd4b75b03b21","Type":"ContainerDied","Data":"f5844fb878905ba1bd55afbd3dcb8b628349954934d71a6c7dfe13db677d7f06"}
Sep 08 13:20:50.240310 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:50.240039    2558 scope.go:117] "RemoveContainer" containerID="2fc5f6f545e6e3ca0e3c62b5b2d8121fc48e876f8fbb8788acdf2cbdb435eaff"
Sep 08 13:20:50.248413 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:50.248394    2558 scope.go:117] "RemoveContainer" containerID="2fc5f6f545e6e3ca0e3c62b5b2d8121fc48e876f8fbb8788acdf2cbdb435eaff"
Sep 08 13:20:50.248702 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:20:50.248683    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fc5f6f545e6e3ca0e3c62b5b2d8121fc48e876f8fbb8788acdf2cbdb435eaff\": container with ID starting with 2fc5f6f545e6e3ca0e3c62b5b2d8121fc48e876f8fbb8788acdf2cbdb435eaff not found: ID does not exist" containerID="2fc5f6f545e6e3ca0e3c62b5b2d8121fc48e876f8fbb8788acdf2cbdb435eaff"
Sep 08 13:20:50.248773 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:50.248709    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc5f6f545e6e3ca0e3c62b5b2d8121fc48e876f8fbb8788acdf2cbdb435eaff"} err="failed to get container status \"2fc5f6f545e6e3ca0e3c62b5b2d8121fc48e876f8fbb8788acdf2cbdb435eaff\": rpc error: code = NotFound desc = could not find container \"2fc5f6f545e6e3ca0e3c62b5b2d8121fc48e876f8fbb8788acdf2cbdb435eaff\": container with ID starting with 2fc5f6f545e6e3ca0e3c62b5b2d8121fc48e876f8fbb8788acdf2cbdb435eaff not found: ID does not exist"
Sep 08 13:20:50.261223 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:50.261202    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65c898f758-b2xjj"]
Sep 08 13:20:50.264760 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:50.264741    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65c898f758-b2xjj"]
Sep 08 13:20:51.052835 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:20:51.052803    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e47f6c2a-615f-4357-beca-dd4b75b03b21" path="/var/lib/kubelet/pods/e47f6c2a-615f-4357-beca-dd4b75b03b21/volumes"
Sep 08 13:22:31.630930 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:31.630890    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-w8rck"]
Sep 08 13:22:31.631491 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:31.631341    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e47f6c2a-615f-4357-beca-dd4b75b03b21" containerName="console"
Sep 08 13:22:31.631491 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:31.631361    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e47f6c2a-615f-4357-beca-dd4b75b03b21" containerName="console"
Sep 08 13:22:31.631491 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:31.631459    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e47f6c2a-615f-4357-beca-dd4b75b03b21" containerName="console"
Sep 08 13:22:31.634363 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:31.634341    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-w8rck"
Sep 08 13:22:31.636646 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:31.636608    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 08 13:22:31.636773 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:31.636654    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 08 13:22:31.636773 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:31.636668    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-4vlbk\""
Sep 08 13:22:31.636959 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:31.636942    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 08 13:22:31.647794 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:31.647766    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-w8rck"]
Sep 08 13:22:31.738455 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:31.738413    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/4786e2ab-6feb-4511-984e-4ee34bc75a27-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-w8rck\" (UID: \"4786e2ab-6feb-4511-984e-4ee34bc75a27\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-w8rck"
Sep 08 13:22:31.738455 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:31.738455    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npl9s\" (UniqueName: \"kubernetes.io/projected/4786e2ab-6feb-4511-984e-4ee34bc75a27-kube-api-access-npl9s\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-w8rck\" (UID: \"4786e2ab-6feb-4511-984e-4ee34bc75a27\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-w8rck"
Sep 08 13:22:31.839440 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:31.839379    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/4786e2ab-6feb-4511-984e-4ee34bc75a27-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-w8rck\" (UID: \"4786e2ab-6feb-4511-984e-4ee34bc75a27\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-w8rck"
Sep 08 13:22:31.839440 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:31.839440    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npl9s\" (UniqueName: \"kubernetes.io/projected/4786e2ab-6feb-4511-984e-4ee34bc75a27-kube-api-access-npl9s\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-w8rck\" (UID: \"4786e2ab-6feb-4511-984e-4ee34bc75a27\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-w8rck"
Sep 08 13:22:31.841774 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:31.841743    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/4786e2ab-6feb-4511-984e-4ee34bc75a27-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-w8rck\" (UID: \"4786e2ab-6feb-4511-984e-4ee34bc75a27\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-w8rck"
Sep 08 13:22:31.848755 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:31.848732    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npl9s\" (UniqueName: \"kubernetes.io/projected/4786e2ab-6feb-4511-984e-4ee34bc75a27-kube-api-access-npl9s\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-w8rck\" (UID: \"4786e2ab-6feb-4511-984e-4ee34bc75a27\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-w8rck"
Sep 08 13:22:31.945330 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:31.945242    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-w8rck"
Sep 08 13:22:32.221465 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:32.221436    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-w8rck"]
Sep 08 13:22:32.224402 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:22:32.224371    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4786e2ab_6feb_4511_984e_4ee34bc75a27.slice/crio-fffc89c0bf4dab9e65fe043e93d437b4fc7be74f0c9069585e5668082e4660a3 WatchSource:0}: Error finding container fffc89c0bf4dab9e65fe043e93d437b4fc7be74f0c9069585e5668082e4660a3: Status 404 returned error can't find the container with id fffc89c0bf4dab9e65fe043e93d437b4fc7be74f0c9069585e5668082e4660a3
Sep 08 13:22:32.226374 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:32.226356    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:22:32.650380 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:32.650340    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-w8rck" event={"ID":"4786e2ab-6feb-4511-984e-4ee34bc75a27","Type":"ContainerStarted","Data":"fffc89c0bf4dab9e65fe043e93d437b4fc7be74f0c9069585e5668082e4660a3"}
Sep 08 13:22:35.667693 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:35.667636    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-w8rck" event={"ID":"4786e2ab-6feb-4511-984e-4ee34bc75a27","Type":"ContainerStarted","Data":"3b3864a115fec733bd08bbc5ffbd00cc283ba0ed020f90871bfd06ef7537fd62"}
Sep 08 13:22:35.668069 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:35.667805    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-w8rck"
Sep 08 13:22:35.691941 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:35.691894    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-w8rck" podStartSLOduration=1.621746476 podStartE2EDuration="4.691879646s" totalImagesPullingTime="3.07013317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:22:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:22:35.690383424 +0000 UTC m=+419.142144366" watchObservedRunningTime="2026-09-08 13:22:35.691879646 +0000 UTC m=+419.143640612"
Sep 08 13:22:36.071739 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:36.071652    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-d6npb"]
Sep 08 13:22:36.074940 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:36.074923    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-d6npb"
Sep 08 13:22:36.077145 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:36.077120    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 08 13:22:36.077320 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:36.077296    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 08 13:22:36.077371 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:36.077348    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-nsq54\""
Sep 08 13:22:36.093824 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:36.093796    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-d6npb"]
Sep 08 13:22:36.175245 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:36.175207    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/2ad3266c-aae8-4b29-b81c-4742656d92e3-cabundle0\") pod \"keda-operator-579c8958f7-d6npb\" (UID: \"2ad3266c-aae8-4b29-b81c-4742656d92e3\") " pod="openshift-keda/keda-operator-579c8958f7-d6npb"
Sep 08 13:22:36.175415 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:36.175341    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2ad3266c-aae8-4b29-b81c-4742656d92e3-certificates\") pod \"keda-operator-579c8958f7-d6npb\" (UID: \"2ad3266c-aae8-4b29-b81c-4742656d92e3\") " pod="openshift-keda/keda-operator-579c8958f7-d6npb"
Sep 08 13:22:36.175415 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:36.175379    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvp22\" (UniqueName: \"kubernetes.io/projected/2ad3266c-aae8-4b29-b81c-4742656d92e3-kube-api-access-jvp22\") pod \"keda-operator-579c8958f7-d6npb\" (UID: \"2ad3266c-aae8-4b29-b81c-4742656d92e3\") " pod="openshift-keda/keda-operator-579c8958f7-d6npb"
Sep 08 13:22:36.276628 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:36.276576    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2ad3266c-aae8-4b29-b81c-4742656d92e3-certificates\") pod \"keda-operator-579c8958f7-d6npb\" (UID: \"2ad3266c-aae8-4b29-b81c-4742656d92e3\") " pod="openshift-keda/keda-operator-579c8958f7-d6npb"
Sep 08 13:22:36.276628 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:36.276635    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvp22\" (UniqueName: \"kubernetes.io/projected/2ad3266c-aae8-4b29-b81c-4742656d92e3-kube-api-access-jvp22\") pod \"keda-operator-579c8958f7-d6npb\" (UID: \"2ad3266c-aae8-4b29-b81c-4742656d92e3\") " pod="openshift-keda/keda-operator-579c8958f7-d6npb"
Sep 08 13:22:36.276881 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:36.276667    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/2ad3266c-aae8-4b29-b81c-4742656d92e3-cabundle0\") pod \"keda-operator-579c8958f7-d6npb\" (UID: \"2ad3266c-aae8-4b29-b81c-4742656d92e3\") " pod="openshift-keda/keda-operator-579c8958f7-d6npb"
Sep 08 13:22:36.276881 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:22:36.276749    2558 secret.go:281] references non-existent secret key: ca.crt
Sep 08 13:22:36.276881 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:22:36.276777    2558 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 08 13:22:36.276881 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:22:36.276789    2558 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-d6npb: references non-existent secret key: ca.crt
Sep 08 13:22:36.276881 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:22:36.276859    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ad3266c-aae8-4b29-b81c-4742656d92e3-certificates podName:2ad3266c-aae8-4b29-b81c-4742656d92e3 nodeName:}" failed. No retries permitted until 2026-09-08 13:22:36.776834005 +0000 UTC m=+420.228594931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/2ad3266c-aae8-4b29-b81c-4742656d92e3-certificates") pod "keda-operator-579c8958f7-d6npb" (UID: "2ad3266c-aae8-4b29-b81c-4742656d92e3") : references non-existent secret key: ca.crt
Sep 08 13:22:36.277303 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:36.277283    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/2ad3266c-aae8-4b29-b81c-4742656d92e3-cabundle0\") pod \"keda-operator-579c8958f7-d6npb\" (UID: \"2ad3266c-aae8-4b29-b81c-4742656d92e3\") " pod="openshift-keda/keda-operator-579c8958f7-d6npb"
Sep 08 13:22:36.285452 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:36.285432    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvp22\" (UniqueName: \"kubernetes.io/projected/2ad3266c-aae8-4b29-b81c-4742656d92e3-kube-api-access-jvp22\") pod \"keda-operator-579c8958f7-d6npb\" (UID: \"2ad3266c-aae8-4b29-b81c-4742656d92e3\") " pod="openshift-keda/keda-operator-579c8958f7-d6npb"
Sep 08 13:22:36.781901 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:36.781869    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2ad3266c-aae8-4b29-b81c-4742656d92e3-certificates\") pod \"keda-operator-579c8958f7-d6npb\" (UID: \"2ad3266c-aae8-4b29-b81c-4742656d92e3\") " pod="openshift-keda/keda-operator-579c8958f7-d6npb"
Sep 08 13:22:36.782341 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:22:36.782032    2558 secret.go:281] references non-existent secret key: ca.crt
Sep 08 13:22:36.782341 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:22:36.782059    2558 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 08 13:22:36.782341 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:22:36.782070    2558 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-d6npb: references non-existent secret key: ca.crt
Sep 08 13:22:36.782341 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:22:36.782133    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ad3266c-aae8-4b29-b81c-4742656d92e3-certificates podName:2ad3266c-aae8-4b29-b81c-4742656d92e3 nodeName:}" failed. No retries permitted until 2026-09-08 13:22:37.782113928 +0000 UTC m=+421.233874858 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/2ad3266c-aae8-4b29-b81c-4742656d92e3-certificates") pod "keda-operator-579c8958f7-d6npb" (UID: "2ad3266c-aae8-4b29-b81c-4742656d92e3") : references non-existent secret key: ca.crt
Sep 08 13:22:37.027545 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:37.027505    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-ffmdf"]
Sep 08 13:22:37.030922 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:37.030905    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-ffmdf"
Sep 08 13:22:37.033631 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:37.033562    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 08 13:22:37.044423 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:37.044401    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-ffmdf"]
Sep 08 13:22:37.085481 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:37.085452    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/87e2e7b1-7601-4b54-9f14-9e09a1807e58-certificates\") pod \"keda-admission-6c4d864b86-ffmdf\" (UID: \"87e2e7b1-7601-4b54-9f14-9e09a1807e58\") " pod="openshift-keda/keda-admission-6c4d864b86-ffmdf"
Sep 08 13:22:37.085620 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:37.085560    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ppgr\" (UniqueName: \"kubernetes.io/projected/87e2e7b1-7601-4b54-9f14-9e09a1807e58-kube-api-access-5ppgr\") pod \"keda-admission-6c4d864b86-ffmdf\" (UID: \"87e2e7b1-7601-4b54-9f14-9e09a1807e58\") " pod="openshift-keda/keda-admission-6c4d864b86-ffmdf"
Sep 08 13:22:37.186967 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:37.186921    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ppgr\" (UniqueName: \"kubernetes.io/projected/87e2e7b1-7601-4b54-9f14-9e09a1807e58-kube-api-access-5ppgr\") pod \"keda-admission-6c4d864b86-ffmdf\" (UID: \"87e2e7b1-7601-4b54-9f14-9e09a1807e58\") " pod="openshift-keda/keda-admission-6c4d864b86-ffmdf"
Sep 08 13:22:37.187158 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:37.187026    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/87e2e7b1-7601-4b54-9f14-9e09a1807e58-certificates\") pod \"keda-admission-6c4d864b86-ffmdf\" (UID: \"87e2e7b1-7601-4b54-9f14-9e09a1807e58\") " pod="openshift-keda/keda-admission-6c4d864b86-ffmdf"
Sep 08 13:22:37.189557 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:37.189525    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/87e2e7b1-7601-4b54-9f14-9e09a1807e58-certificates\") pod \"keda-admission-6c4d864b86-ffmdf\" (UID: \"87e2e7b1-7601-4b54-9f14-9e09a1807e58\") " pod="openshift-keda/keda-admission-6c4d864b86-ffmdf"
Sep 08 13:22:37.195497 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:37.195476    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ppgr\" (UniqueName: \"kubernetes.io/projected/87e2e7b1-7601-4b54-9f14-9e09a1807e58-kube-api-access-5ppgr\") pod \"keda-admission-6c4d864b86-ffmdf\" (UID: \"87e2e7b1-7601-4b54-9f14-9e09a1807e58\") " pod="openshift-keda/keda-admission-6c4d864b86-ffmdf"
Sep 08 13:22:37.343986 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:37.343897    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-nsq54\""
Sep 08 13:22:37.351917 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:37.351885    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-ffmdf"
Sep 08 13:22:37.630732 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:37.630706    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-ffmdf"]
Sep 08 13:22:37.632814 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:22:37.632788    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e2e7b1_7601_4b54_9f14_9e09a1807e58.slice/crio-59fc949bb7d73d53ecd636923b6fed94a30a7a57bf5d363726f4fcf9ab250d9b WatchSource:0}: Error finding container 59fc949bb7d73d53ecd636923b6fed94a30a7a57bf5d363726f4fcf9ab250d9b: Status 404 returned error can't find the container with id 59fc949bb7d73d53ecd636923b6fed94a30a7a57bf5d363726f4fcf9ab250d9b
Sep 08 13:22:37.678878 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:37.678843    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-ffmdf" event={"ID":"87e2e7b1-7601-4b54-9f14-9e09a1807e58","Type":"ContainerStarted","Data":"59fc949bb7d73d53ecd636923b6fed94a30a7a57bf5d363726f4fcf9ab250d9b"}
Sep 08 13:22:37.798121 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:37.798083    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2ad3266c-aae8-4b29-b81c-4742656d92e3-certificates\") pod \"keda-operator-579c8958f7-d6npb\" (UID: \"2ad3266c-aae8-4b29-b81c-4742656d92e3\") " pod="openshift-keda/keda-operator-579c8958f7-d6npb"
Sep 08 13:22:37.800433 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:37.800413    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2ad3266c-aae8-4b29-b81c-4742656d92e3-certificates\") pod \"keda-operator-579c8958f7-d6npb\" (UID: \"2ad3266c-aae8-4b29-b81c-4742656d92e3\") " pod="openshift-keda/keda-operator-579c8958f7-d6npb"
Sep 08 13:22:37.886163 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:37.886071    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-d6npb"
Sep 08 13:22:38.191052 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:38.191022    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-d6npb"]
Sep 08 13:22:38.192228 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:22:38.192200    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad3266c_aae8_4b29_b81c_4742656d92e3.slice/crio-86795dece090486bb263e8c6f383b3d68cecfdba143c5a7610c4f693fb70b1fa WatchSource:0}: Error finding container 86795dece090486bb263e8c6f383b3d68cecfdba143c5a7610c4f693fb70b1fa: Status 404 returned error can't find the container with id 86795dece090486bb263e8c6f383b3d68cecfdba143c5a7610c4f693fb70b1fa
Sep 08 13:22:38.684374 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:38.684338    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-d6npb" event={"ID":"2ad3266c-aae8-4b29-b81c-4742656d92e3","Type":"ContainerStarted","Data":"86795dece090486bb263e8c6f383b3d68cecfdba143c5a7610c4f693fb70b1fa"}
Sep 08 13:22:41.701122 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:41.701081    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-ffmdf" event={"ID":"87e2e7b1-7601-4b54-9f14-9e09a1807e58","Type":"ContainerStarted","Data":"492e3e030efd91baaf112374b3143ec0bf46e3cb8dd6f0c00a912e5d5269067b"}
Sep 08 13:22:41.701568 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:41.701204    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-ffmdf"
Sep 08 13:22:41.702808 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:41.702780    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-d6npb" event={"ID":"2ad3266c-aae8-4b29-b81c-4742656d92e3","Type":"ContainerStarted","Data":"4fe2155424888fd9af0ffe4a73c252aeb06045fb73c0b8da169e424b7778ce8b"}
Sep 08 13:22:41.702922 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:41.702837    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-d6npb"
Sep 08 13:22:41.720046 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:41.720004    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-ffmdf" podStartSLOduration=0.769195811 podStartE2EDuration="4.71999041s" totalImagesPullingTime="3.950794599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:22:41.718872339 +0000 UTC m=+425.170633270" watchObservedRunningTime="2026-09-08 13:22:41.71999041 +0000 UTC m=+425.171751340"
Sep 08 13:22:41.737104 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:41.737056    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-d6npb" podStartSLOduration=2.342650851 podStartE2EDuration="5.737040034s" totalImagesPullingTime="3.394389183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:22:41.735796199 +0000 UTC m=+425.187557132" watchObservedRunningTime="2026-09-08 13:22:41.737040034 +0000 UTC m=+425.188800965"
Sep 08 13:22:56.674865 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:22:56.674830    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-w8rck"
Sep 08 13:23:02.709892 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:23:02.709806    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-d6npb"
Sep 08 13:23:02.709892 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:23:02.709885    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-ffmdf"
Sep 08 13:23:51.883712 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:23:51.883678    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c9d476b6c-ksvld"]
Sep 08 13:23:56.151060 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:23:56.151016    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-w2v9h"]
Sep 08 13:23:56.154683 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:23:56.154660    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-w2v9h"
Sep 08 13:23:56.158431 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:23:56.158408    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 08 13:23:56.158431 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:23:56.158426    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-gphgc\""
Sep 08 13:23:56.159023 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:23:56.159007    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 08 13:23:56.165843 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:23:56.165821    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-w2v9h"]
Sep 08 13:23:56.242860 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:23:56.242828    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b8d69677-5433-4f2e-a995-73b48d3ed52c-bound-sa-token\") pod \"cert-manager-55b4d954d5-w2v9h\" (UID: \"b8d69677-5433-4f2e-a995-73b48d3ed52c\") " pod="cert-manager/cert-manager-55b4d954d5-w2v9h"
Sep 08 13:23:56.243032 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:23:56.242897    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfspw\" (UniqueName: \"kubernetes.io/projected/b8d69677-5433-4f2e-a995-73b48d3ed52c-kube-api-access-xfspw\") pod \"cert-manager-55b4d954d5-w2v9h\" (UID: \"b8d69677-5433-4f2e-a995-73b48d3ed52c\") " pod="cert-manager/cert-manager-55b4d954d5-w2v9h"
Sep 08 13:23:56.343560 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:23:56.343521    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfspw\" (UniqueName: \"kubernetes.io/projected/b8d69677-5433-4f2e-a995-73b48d3ed52c-kube-api-access-xfspw\") pod \"cert-manager-55b4d954d5-w2v9h\" (UID: \"b8d69677-5433-4f2e-a995-73b48d3ed52c\") " pod="cert-manager/cert-manager-55b4d954d5-w2v9h"
Sep 08 13:23:56.343758 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:23:56.343689    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b8d69677-5433-4f2e-a995-73b48d3ed52c-bound-sa-token\") pod \"cert-manager-55b4d954d5-w2v9h\" (UID: \"b8d69677-5433-4f2e-a995-73b48d3ed52c\") " pod="cert-manager/cert-manager-55b4d954d5-w2v9h"
Sep 08 13:23:56.351951 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:23:56.351922    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b8d69677-5433-4f2e-a995-73b48d3ed52c-bound-sa-token\") pod \"cert-manager-55b4d954d5-w2v9h\" (UID: \"b8d69677-5433-4f2e-a995-73b48d3ed52c\") " pod="cert-manager/cert-manager-55b4d954d5-w2v9h"
Sep 08 13:23:56.352058 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:23:56.351960    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfspw\" (UniqueName: \"kubernetes.io/projected/b8d69677-5433-4f2e-a995-73b48d3ed52c-kube-api-access-xfspw\") pod \"cert-manager-55b4d954d5-w2v9h\" (UID: \"b8d69677-5433-4f2e-a995-73b48d3ed52c\") " pod="cert-manager/cert-manager-55b4d954d5-w2v9h"
Sep 08 13:23:56.470529 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:23:56.470434    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-w2v9h"
Sep 08 13:23:56.741159 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:23:56.741133    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-w2v9h"]
Sep 08 13:23:56.743850 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:23:56.743819    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8d69677_5433_4f2e_a995_73b48d3ed52c.slice/crio-cd19ba002c9d571851d4296f7b046b06d329dcf673a439452e65cc086793bf04 WatchSource:0}: Error finding container cd19ba002c9d571851d4296f7b046b06d329dcf673a439452e65cc086793bf04: Status 404 returned error can't find the container with id cd19ba002c9d571851d4296f7b046b06d329dcf673a439452e65cc086793bf04
Sep 08 13:23:57.035265 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:23:57.035184    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-w2v9h" event={"ID":"b8d69677-5433-4f2e-a995-73b48d3ed52c","Type":"ContainerStarted","Data":"cd19ba002c9d571851d4296f7b046b06d329dcf673a439452e65cc086793bf04"}
Sep 08 13:24:00.052561 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:00.052526    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-w2v9h" event={"ID":"b8d69677-5433-4f2e-a995-73b48d3ed52c","Type":"ContainerStarted","Data":"1c1549ec3e5a5453396d227032e69ec6e33de0bcbf6d961f72913e572db0aa58"}
Sep 08 13:24:00.068468 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:00.068424    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-w2v9h" podStartSLOduration=0.926933544 podStartE2EDuration="4.068406861s" totalImagesPullingTime="3.141473317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:23:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:24:00.066765891 +0000 UTC m=+503.518526822" watchObservedRunningTime="2026-09-08 13:24:00.068406861 +0000 UTC m=+503.520167793"
Sep 08 13:24:14.956004 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:14.955963    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzjrd"]
Sep 08 13:24:14.959463 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:14.959440    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzjrd"
Sep 08 13:24:14.963051 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:14.963028    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:24:14.963172 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:14.963096    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 08 13:24:14.963846 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:14.963825    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-gxfpq\""
Sep 08 13:24:14.980565 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:14.980539    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzjrd"]
Sep 08 13:24:15.009900 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:15.009871    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fa216db-06cc-4816-8010-a7bb50bb3a88-tmp\") pod \"openshift-lws-operator-fd8ccff4c-kzjrd\" (UID: \"0fa216db-06cc-4816-8010-a7bb50bb3a88\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzjrd"
Sep 08 13:24:15.010091 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:15.009977    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5knm9\" (UniqueName: \"kubernetes.io/projected/0fa216db-06cc-4816-8010-a7bb50bb3a88-kube-api-access-5knm9\") pod \"openshift-lws-operator-fd8ccff4c-kzjrd\" (UID: \"0fa216db-06cc-4816-8010-a7bb50bb3a88\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzjrd"
Sep 08 13:24:15.110439 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:15.110398    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fa216db-06cc-4816-8010-a7bb50bb3a88-tmp\") pod \"openshift-lws-operator-fd8ccff4c-kzjrd\" (UID: \"0fa216db-06cc-4816-8010-a7bb50bb3a88\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzjrd"
Sep 08 13:24:15.110673 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:15.110498    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5knm9\" (UniqueName: \"kubernetes.io/projected/0fa216db-06cc-4816-8010-a7bb50bb3a88-kube-api-access-5knm9\") pod \"openshift-lws-operator-fd8ccff4c-kzjrd\" (UID: \"0fa216db-06cc-4816-8010-a7bb50bb3a88\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzjrd"
Sep 08 13:24:15.110870 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:15.110844    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fa216db-06cc-4816-8010-a7bb50bb3a88-tmp\") pod \"openshift-lws-operator-fd8ccff4c-kzjrd\" (UID: \"0fa216db-06cc-4816-8010-a7bb50bb3a88\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzjrd"
Sep 08 13:24:15.128070 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:15.128037    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5knm9\" (UniqueName: \"kubernetes.io/projected/0fa216db-06cc-4816-8010-a7bb50bb3a88-kube-api-access-5knm9\") pod \"openshift-lws-operator-fd8ccff4c-kzjrd\" (UID: \"0fa216db-06cc-4816-8010-a7bb50bb3a88\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzjrd"
Sep 08 13:24:15.269236 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:15.269131    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzjrd"
Sep 08 13:24:15.552815 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:15.552740    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzjrd"]
Sep 08 13:24:15.556107 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:24:15.556076    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fa216db_06cc_4816_8010_a7bb50bb3a88.slice/crio-473ce88f093e4cba0bdbe84fae5ff2628cb183128d6a00479a2292fb73184571 WatchSource:0}: Error finding container 473ce88f093e4cba0bdbe84fae5ff2628cb183128d6a00479a2292fb73184571: Status 404 returned error can't find the container with id 473ce88f093e4cba0bdbe84fae5ff2628cb183128d6a00479a2292fb73184571
Sep 08 13:24:16.123943 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:16.123888    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzjrd" event={"ID":"0fa216db-06cc-4816-8010-a7bb50bb3a88","Type":"ContainerStarted","Data":"473ce88f093e4cba0bdbe84fae5ff2628cb183128d6a00479a2292fb73184571"}
Sep 08 13:24:16.903377 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:16.903306    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7c9d476b6c-ksvld" podUID="276be51b-30a3-468b-be17-7a67a4654752" containerName="console" containerID="cri-o://c8c770a4998d6813e25abadee7692dccc9963be2e14d79139565a67318465339" gracePeriod=15
Sep 08 13:24:17.965544 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:17.965512    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9d476b6c-ksvld"
Sep 08 13:24:18.034855 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:18.034825    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/276be51b-30a3-468b-be17-7a67a4654752-console-serving-cert\") pod \"276be51b-30a3-468b-be17-7a67a4654752\" (UID: \"276be51b-30a3-468b-be17-7a67a4654752\") "
Sep 08 13:24:18.035047 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:18.034874    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/276be51b-30a3-468b-be17-7a67a4654752-console-config\") pod \"276be51b-30a3-468b-be17-7a67a4654752\" (UID: \"276be51b-30a3-468b-be17-7a67a4654752\") "
Sep 08 13:24:18.035047 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:18.034896    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/276be51b-30a3-468b-be17-7a67a4654752-trusted-ca-bundle\") pod \"276be51b-30a3-468b-be17-7a67a4654752\" (UID: \"276be51b-30a3-468b-be17-7a67a4654752\") "
Sep 08 13:24:18.035047 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:18.034924    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/276be51b-30a3-468b-be17-7a67a4654752-console-oauth-config\") pod \"276be51b-30a3-468b-be17-7a67a4654752\" (UID: \"276be51b-30a3-468b-be17-7a67a4654752\") "
Sep 08 13:24:18.035047 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:18.034955    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/276be51b-30a3-468b-be17-7a67a4654752-service-ca\") pod \"276be51b-30a3-468b-be17-7a67a4654752\" (UID: \"276be51b-30a3-468b-be17-7a67a4654752\") "
Sep 08 13:24:18.035047 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:18.034977    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/276be51b-30a3-468b-be17-7a67a4654752-oauth-serving-cert\") pod \"276be51b-30a3-468b-be17-7a67a4654752\" (UID: \"276be51b-30a3-468b-be17-7a67a4654752\") "
Sep 08 13:24:18.035047 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:18.035012    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzbt6\" (UniqueName: \"kubernetes.io/projected/276be51b-30a3-468b-be17-7a67a4654752-kube-api-access-jzbt6\") pod \"276be51b-30a3-468b-be17-7a67a4654752\" (UID: \"276be51b-30a3-468b-be17-7a67a4654752\") "
Sep 08 13:24:18.035362 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:18.035226    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/276be51b-30a3-468b-be17-7a67a4654752-console-config" (OuterVolumeSpecName: "console-config") pod "276be51b-30a3-468b-be17-7a67a4654752" (UID: "276be51b-30a3-468b-be17-7a67a4654752"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:24:18.035467 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:18.035442    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/276be51b-30a3-468b-be17-7a67a4654752-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "276be51b-30a3-468b-be17-7a67a4654752" (UID: "276be51b-30a3-468b-be17-7a67a4654752"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:24:18.035538 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:18.035479    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/276be51b-30a3-468b-be17-7a67a4654752-service-ca" (OuterVolumeSpecName: "service-ca") pod "276be51b-30a3-468b-be17-7a67a4654752" (UID: "276be51b-30a3-468b-be17-7a67a4654752"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:24:18.035630 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:18.035580    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/276be51b-30a3-468b-be17-7a67a4654752-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "276be51b-30a3-468b-be17-7a67a4654752" (UID: "276be51b-30a3-468b-be17-7a67a4654752"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:24:18.037389 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:18.037362    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/276be51b-30a3-468b-be17-7a67a4654752-kube-api-access-jzbt6" (OuterVolumeSpecName: "kube-api-access-jzbt6") pod "276be51b-30a3-468b-be17-7a67a4654752" (UID: "276be51b-30a3-468b-be17-7a67a4654752"). InnerVolumeSpecName "kube-api-access-jzbt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:24:18.037805 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:18.037778    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/276be51b-30a3-468b-be17-7a67a4654752-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "276be51b-30a3-468b-be17-7a67a4654752" (UID: "276be51b-30a3-468b-be17-7a67a4654752"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:24:18.037805 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:18.037795    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/276be51b-30a3-468b-be17-7a67a4654752-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "276be51b-30a3-468b-be17-7a67a4654752" (UID: "276be51b-30a3-468b-be17-7a67a4654752"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:24:18.135220 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:18.135180    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzjrd" event={"ID":"0fa216db-06cc-4816-8010-a7bb50bb3a88","Type":"ContainerStarted","Data":"ac7db9bcd105d865381d467c7158aa4552655941ab4487be39478094bfabfb0b"}
Sep 08 13:24:18.135667 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:18.135635    2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/276be51b-30a3-468b-be17-7a67a4654752-service-ca\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:24:18.135667 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:18.135667    2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/276be51b-30a3-468b-be17-7a67a4654752-oauth-serving-cert\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:24:18.135842 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:18.135682    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzbt6\" (UniqueName: \"kubernetes.io/projected/276be51b-30a3-468b-be17-7a67a4654752-kube-api-access-jzbt6\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:24:18.135842 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:18.135695    2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/276be51b-30a3-468b-be17-7a67a4654752-console-serving-cert\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:24:18.135842 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:18.135707    2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/276be51b-30a3-468b-be17-7a67a4654752-console-config\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:24:18.135842 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:18.135718    2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/276be51b-30a3-468b-be17-7a67a4654752-trusted-ca-bundle\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:24:18.135842 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:18.135729    2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/276be51b-30a3-468b-be17-7a67a4654752-console-oauth-config\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:24:18.136628 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:18.136585    2558 generic.go:356] "Generic (PLEG): container finished" podID="276be51b-30a3-468b-be17-7a67a4654752" containerID="c8c770a4998d6813e25abadee7692dccc9963be2e14d79139565a67318465339" exitCode=0
Sep 08 13:24:18.136715 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:18.136643    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9d476b6c-ksvld"
Sep 08 13:24:18.136715 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:18.136684    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9d476b6c-ksvld" event={"ID":"276be51b-30a3-468b-be17-7a67a4654752","Type":"ContainerDied","Data":"c8c770a4998d6813e25abadee7692dccc9963be2e14d79139565a67318465339"}
Sep 08 13:24:18.136779 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:18.136737    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9d476b6c-ksvld" event={"ID":"276be51b-30a3-468b-be17-7a67a4654752","Type":"ContainerDied","Data":"2c2b6a8c7f895644b657cfe8844e1a80a55b8b9a85bbd76b0072afb7530dfe32"}
Sep 08 13:24:18.136779 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:18.136765    2558 scope.go:117] "RemoveContainer" containerID="c8c770a4998d6813e25abadee7692dccc9963be2e14d79139565a67318465339"
Sep 08 13:24:18.146075 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:18.146055    2558 scope.go:117] "RemoveContainer" containerID="c8c770a4998d6813e25abadee7692dccc9963be2e14d79139565a67318465339"
Sep 08 13:24:18.146345 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:24:18.146328    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8c770a4998d6813e25abadee7692dccc9963be2e14d79139565a67318465339\": container with ID starting with c8c770a4998d6813e25abadee7692dccc9963be2e14d79139565a67318465339 not found: ID does not exist" containerID="c8c770a4998d6813e25abadee7692dccc9963be2e14d79139565a67318465339"
Sep 08 13:24:18.146398 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:18.146353    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8c770a4998d6813e25abadee7692dccc9963be2e14d79139565a67318465339"} err="failed to get container status \"c8c770a4998d6813e25abadee7692dccc9963be2e14d79139565a67318465339\": rpc error: code = NotFound desc = could not find container \"c8c770a4998d6813e25abadee7692dccc9963be2e14d79139565a67318465339\": container with ID starting with c8c770a4998d6813e25abadee7692dccc9963be2e14d79139565a67318465339 not found: ID does not exist"
Sep 08 13:24:18.164607 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:18.164539    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-kzjrd" podStartSLOduration=1.831855276 podStartE2EDuration="4.164521749s" totalImagesPullingTime="2.332666473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:24:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:24:18.164165589 +0000 UTC m=+521.615926522" watchObservedRunningTime="2026-09-08 13:24:18.164521749 +0000 UTC m=+521.616282680"
Sep 08 13:24:18.183678 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:18.183640    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c9d476b6c-ksvld"]
Sep 08 13:24:18.193004 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:18.192973    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c9d476b6c-ksvld"]
Sep 08 13:24:19.053536 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:19.053502    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="276be51b-30a3-468b-be17-7a67a4654752" path="/var/lib/kubelet/pods/276be51b-30a3-468b-be17-7a67a4654752/volumes"
Sep 08 13:24:28.234914 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:28.234875    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-66c568d8-t6qln"]
Sep 08 13:24:28.235380 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:28.235202    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="276be51b-30a3-468b-be17-7a67a4654752" containerName="console"
Sep 08 13:24:28.235380 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:28.235213    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="276be51b-30a3-468b-be17-7a67a4654752" containerName="console"
Sep 08 13:24:28.235380 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:28.235277    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="276be51b-30a3-468b-be17-7a67a4654752" containerName="console"
Sep 08 13:24:28.243220 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:28.243195    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-66c568d8-t6qln"
Sep 08 13:24:28.247343 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:28.247317    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-hl9k8\""
Sep 08 13:24:28.247470 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:28.247317    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 08 13:24:28.247470 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:28.247461    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 08 13:24:28.247663 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:28.247638    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 08 13:24:28.259119 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:28.259092    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-66c568d8-t6qln"]
Sep 08 13:24:28.322317 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:28.322281    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2d213d1c-5fbd-4f74-881d-9754ee40eb1a-manager-config\") pod \"lws-controller-manager-66c568d8-t6qln\" (UID: \"2d213d1c-5fbd-4f74-881d-9754ee40eb1a\") " pod="openshift-lws-operator/lws-controller-manager-66c568d8-t6qln"
Sep 08 13:24:28.322317 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:28.322319    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ph8w\" (UniqueName: \"kubernetes.io/projected/2d213d1c-5fbd-4f74-881d-9754ee40eb1a-kube-api-access-2ph8w\") pod \"lws-controller-manager-66c568d8-t6qln\" (UID: \"2d213d1c-5fbd-4f74-881d-9754ee40eb1a\") " pod="openshift-lws-operator/lws-controller-manager-66c568d8-t6qln"
Sep 08 13:24:28.322521 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:28.322339    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d213d1c-5fbd-4f74-881d-9754ee40eb1a-metrics-cert\") pod \"lws-controller-manager-66c568d8-t6qln\" (UID: \"2d213d1c-5fbd-4f74-881d-9754ee40eb1a\") " pod="openshift-lws-operator/lws-controller-manager-66c568d8-t6qln"
Sep 08 13:24:28.322521 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:28.322416    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d213d1c-5fbd-4f74-881d-9754ee40eb1a-cert\") pod \"lws-controller-manager-66c568d8-t6qln\" (UID: \"2d213d1c-5fbd-4f74-881d-9754ee40eb1a\") " pod="openshift-lws-operator/lws-controller-manager-66c568d8-t6qln"
Sep 08 13:24:28.422905 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:28.422860    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2d213d1c-5fbd-4f74-881d-9754ee40eb1a-manager-config\") pod \"lws-controller-manager-66c568d8-t6qln\" (UID: \"2d213d1c-5fbd-4f74-881d-9754ee40eb1a\") " pod="openshift-lws-operator/lws-controller-manager-66c568d8-t6qln"
Sep 08 13:24:28.422905 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:28.422901    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ph8w\" (UniqueName: \"kubernetes.io/projected/2d213d1c-5fbd-4f74-881d-9754ee40eb1a-kube-api-access-2ph8w\") pod \"lws-controller-manager-66c568d8-t6qln\" (UID: \"2d213d1c-5fbd-4f74-881d-9754ee40eb1a\") " pod="openshift-lws-operator/lws-controller-manager-66c568d8-t6qln"
Sep 08 13:24:28.423171 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:28.422920    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d213d1c-5fbd-4f74-881d-9754ee40eb1a-metrics-cert\") pod \"lws-controller-manager-66c568d8-t6qln\" (UID: \"2d213d1c-5fbd-4f74-881d-9754ee40eb1a\") " pod="openshift-lws-operator/lws-controller-manager-66c568d8-t6qln"
Sep 08 13:24:28.423171 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:28.422942    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d213d1c-5fbd-4f74-881d-9754ee40eb1a-cert\") pod \"lws-controller-manager-66c568d8-t6qln\" (UID: \"2d213d1c-5fbd-4f74-881d-9754ee40eb1a\") " pod="openshift-lws-operator/lws-controller-manager-66c568d8-t6qln"
Sep 08 13:24:28.423667 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:28.423639    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2d213d1c-5fbd-4f74-881d-9754ee40eb1a-manager-config\") pod \"lws-controller-manager-66c568d8-t6qln\" (UID: \"2d213d1c-5fbd-4f74-881d-9754ee40eb1a\") " pod="openshift-lws-operator/lws-controller-manager-66c568d8-t6qln"
Sep 08 13:24:28.425491 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:28.425464    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d213d1c-5fbd-4f74-881d-9754ee40eb1a-cert\") pod \"lws-controller-manager-66c568d8-t6qln\" (UID: \"2d213d1c-5fbd-4f74-881d-9754ee40eb1a\") " pod="openshift-lws-operator/lws-controller-manager-66c568d8-t6qln"
Sep 08 13:24:28.425695 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:28.425678    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d213d1c-5fbd-4f74-881d-9754ee40eb1a-metrics-cert\") pod \"lws-controller-manager-66c568d8-t6qln\" (UID: \"2d213d1c-5fbd-4f74-881d-9754ee40eb1a\") " pod="openshift-lws-operator/lws-controller-manager-66c568d8-t6qln"
Sep 08 13:24:28.487840 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:28.487766    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ph8w\" (UniqueName: \"kubernetes.io/projected/2d213d1c-5fbd-4f74-881d-9754ee40eb1a-kube-api-access-2ph8w\") pod \"lws-controller-manager-66c568d8-t6qln\" (UID: \"2d213d1c-5fbd-4f74-881d-9754ee40eb1a\") " pod="openshift-lws-operator/lws-controller-manager-66c568d8-t6qln"
Sep 08 13:24:28.552252 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:28.552212    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-66c568d8-t6qln"
Sep 08 13:24:28.847198 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:28.847119    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-66c568d8-t6qln"]
Sep 08 13:24:28.850696 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:24:28.850666    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d213d1c_5fbd_4f74_881d_9754ee40eb1a.slice/crio-3aab2d94463b4b50c634bbcc5d8fc0cd70884e5f28269895d85c2035bedcf3e7 WatchSource:0}: Error finding container 3aab2d94463b4b50c634bbcc5d8fc0cd70884e5f28269895d85c2035bedcf3e7: Status 404 returned error can't find the container with id 3aab2d94463b4b50c634bbcc5d8fc0cd70884e5f28269895d85c2035bedcf3e7
Sep 08 13:24:29.188345 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:29.188307    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-66c568d8-t6qln" event={"ID":"2d213d1c-5fbd-4f74-881d-9754ee40eb1a","Type":"ContainerStarted","Data":"3aab2d94463b4b50c634bbcc5d8fc0cd70884e5f28269895d85c2035bedcf3e7"}
Sep 08 13:24:31.198997 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:31.198962    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-66c568d8-t6qln" event={"ID":"2d213d1c-5fbd-4f74-881d-9754ee40eb1a","Type":"ContainerStarted","Data":"3d8eb5780fb3a1f61a69c6daa455aefc4a2c4adc550f26f8ad8f86b9c7b65f57"}
Sep 08 13:24:31.199384 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:31.199101    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-66c568d8-t6qln"
Sep 08 13:24:42.207580 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:42.207536    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-66c568d8-t6qln"
Sep 08 13:24:42.224687 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:24:42.224632    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-66c568d8-t6qln" podStartSLOduration=12.632470103 podStartE2EDuration="14.224580234s" totalImagesPullingTime="1.592110131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:24:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:24:31.23745378 +0000 UTC m=+534.689214713" watchObservedRunningTime="2026-09-08 13:24:42.224580234 +0000 UTC m=+545.676341164"
Sep 08 13:25:21.409576 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:25:21.409538    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-r4kmc"]
Sep 08 13:25:21.413842 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:25:21.413819    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-r4kmc"
Sep 08 13:25:21.416330 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:25:21.416305    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nlm5f\""
Sep 08 13:25:21.416701 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:25:21.416683    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 08 13:25:21.417239 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:25:21.417222    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 08 13:25:21.417338 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:25:21.417279    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 08 13:25:21.417338 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:25:21.417283    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 08 13:25:21.423981 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:25:21.423956    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-r4kmc"]
Sep 08 13:25:21.472500 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:25:21.472460    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8cc5ec5-fcc5-4d39-903f-95a3afe1f61b-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-r4kmc\" (UID: \"b8cc5ec5-fcc5-4d39-903f-95a3afe1f61b\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-r4kmc"
Sep 08 13:25:21.472677 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:25:21.472511    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx7rb\" (UniqueName: \"kubernetes.io/projected/b8cc5ec5-fcc5-4d39-903f-95a3afe1f61b-kube-api-access-jx7rb\") pod \"kuadrant-console-plugin-598f86468b-r4kmc\" (UID: \"b8cc5ec5-fcc5-4d39-903f-95a3afe1f61b\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-r4kmc"
Sep 08 13:25:21.472677 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:25:21.472546    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b8cc5ec5-fcc5-4d39-903f-95a3afe1f61b-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-r4kmc\" (UID: \"b8cc5ec5-fcc5-4d39-903f-95a3afe1f61b\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-r4kmc"
Sep 08 13:25:21.573832 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:25:21.573792    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8cc5ec5-fcc5-4d39-903f-95a3afe1f61b-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-r4kmc\" (UID: \"b8cc5ec5-fcc5-4d39-903f-95a3afe1f61b\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-r4kmc"
Sep 08 13:25:21.573832 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:25:21.573842    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx7rb\" (UniqueName: \"kubernetes.io/projected/b8cc5ec5-fcc5-4d39-903f-95a3afe1f61b-kube-api-access-jx7rb\") pod \"kuadrant-console-plugin-598f86468b-r4kmc\" (UID: \"b8cc5ec5-fcc5-4d39-903f-95a3afe1f61b\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-r4kmc"
Sep 08 13:25:21.574085 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:25:21.573861    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b8cc5ec5-fcc5-4d39-903f-95a3afe1f61b-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-r4kmc\" (UID: \"b8cc5ec5-fcc5-4d39-903f-95a3afe1f61b\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-r4kmc"
Sep 08 13:25:21.574537 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:25:21.574510    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b8cc5ec5-fcc5-4d39-903f-95a3afe1f61b-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-r4kmc\" (UID: \"b8cc5ec5-fcc5-4d39-903f-95a3afe1f61b\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-r4kmc"
Sep 08 13:25:21.576630 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:25:21.576587    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8cc5ec5-fcc5-4d39-903f-95a3afe1f61b-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-r4kmc\" (UID: \"b8cc5ec5-fcc5-4d39-903f-95a3afe1f61b\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-r4kmc"
Sep 08 13:25:21.583346 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:25:21.583317    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx7rb\" (UniqueName: \"kubernetes.io/projected/b8cc5ec5-fcc5-4d39-903f-95a3afe1f61b-kube-api-access-jx7rb\") pod \"kuadrant-console-plugin-598f86468b-r4kmc\" (UID: \"b8cc5ec5-fcc5-4d39-903f-95a3afe1f61b\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-r4kmc"
Sep 08 13:25:21.724292 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:25:21.724192    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-r4kmc"
Sep 08 13:25:22.011106 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:25:22.011031    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-r4kmc"]
Sep 08 13:25:22.014258 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:25:22.014227    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8cc5ec5_fcc5_4d39_903f_95a3afe1f61b.slice/crio-89ca94bd00a83d9c249f87a82bacf5b3ad8349355e52aa0d687f62973a8f34f5 WatchSource:0}: Error finding container 89ca94bd00a83d9c249f87a82bacf5b3ad8349355e52aa0d687f62973a8f34f5: Status 404 returned error can't find the container with id 89ca94bd00a83d9c249f87a82bacf5b3ad8349355e52aa0d687f62973a8f34f5
Sep 08 13:25:22.429868 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:25:22.429831    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-r4kmc" event={"ID":"b8cc5ec5-fcc5-4d39-903f-95a3afe1f61b","Type":"ContainerStarted","Data":"89ca94bd00a83d9c249f87a82bacf5b3ad8349355e52aa0d687f62973a8f34f5"}
Sep 08 13:25:30.473185 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:25:30.473143    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-r4kmc" event={"ID":"b8cc5ec5-fcc5-4d39-903f-95a3afe1f61b","Type":"ContainerStarted","Data":"8dbbcd238e68aa5611d16af66f4fe38405d16f2daf4416d32d82f05a4eb100ad"}
Sep 08 13:25:30.490538 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:25:30.490489    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-r4kmc" podStartSLOduration=1.51264553 podStartE2EDuration="9.490473549s" totalImagesPullingTime="7.977828019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:25:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:25:30.489791732 +0000 UTC m=+593.941552664" watchObservedRunningTime="2026-09-08 13:25:30.490473549 +0000 UTC m=+593.942234480"
Sep 08 13:26:06.699690 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:06.699602    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-fv6z7"]
Sep 08 13:26:06.703012 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:06.702995    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-fv6z7"
Sep 08 13:26:06.705446 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:06.705425    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-n5crc\""
Sep 08 13:26:06.710326 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:06.710070    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-fv6z7"]
Sep 08 13:26:06.727831 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:06.727801    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rqrd\" (UniqueName: \"kubernetes.io/projected/7ce36be3-3649-478c-b536-af51dca1683d-kube-api-access-4rqrd\") pod \"authorino-b54df94b4-fv6z7\" (UID: \"7ce36be3-3649-478c-b536-af51dca1683d\") " pod="kuadrant-system/authorino-b54df94b4-fv6z7"
Sep 08 13:26:06.822270 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:06.822233    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-9wk46"]
Sep 08 13:26:06.825560 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:06.825539    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-9wk46"
Sep 08 13:26:06.828613 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:06.828535    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rqrd\" (UniqueName: \"kubernetes.io/projected/7ce36be3-3649-478c-b536-af51dca1683d-kube-api-access-4rqrd\") pod \"authorino-b54df94b4-fv6z7\" (UID: \"7ce36be3-3649-478c-b536-af51dca1683d\") " pod="kuadrant-system/authorino-b54df94b4-fv6z7"
Sep 08 13:26:06.828754 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:06.828690    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqwbn\" (UniqueName: \"kubernetes.io/projected/e083282d-e401-4e6c-8a92-c3b1200f9341-kube-api-access-hqwbn\") pod \"authorino-c548b5d7d-9wk46\" (UID: \"e083282d-e401-4e6c-8a92-c3b1200f9341\") " pod="kuadrant-system/authorino-c548b5d7d-9wk46"
Sep 08 13:26:06.830273 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:06.830247    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-9wk46"]
Sep 08 13:26:06.842170 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:06.842140    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rqrd\" (UniqueName: \"kubernetes.io/projected/7ce36be3-3649-478c-b536-af51dca1683d-kube-api-access-4rqrd\") pod \"authorino-b54df94b4-fv6z7\" (UID: \"7ce36be3-3649-478c-b536-af51dca1683d\") " pod="kuadrant-system/authorino-b54df94b4-fv6z7"
Sep 08 13:26:06.929095 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:06.929053    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqwbn\" (UniqueName: \"kubernetes.io/projected/e083282d-e401-4e6c-8a92-c3b1200f9341-kube-api-access-hqwbn\") pod \"authorino-c548b5d7d-9wk46\" (UID: \"e083282d-e401-4e6c-8a92-c3b1200f9341\") " pod="kuadrant-system/authorino-c548b5d7d-9wk46"
Sep 08 13:26:06.937835 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:06.937801    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqwbn\" (UniqueName: \"kubernetes.io/projected/e083282d-e401-4e6c-8a92-c3b1200f9341-kube-api-access-hqwbn\") pod \"authorino-c548b5d7d-9wk46\" (UID: \"e083282d-e401-4e6c-8a92-c3b1200f9341\") " pod="kuadrant-system/authorino-c548b5d7d-9wk46"
Sep 08 13:26:07.013054 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:07.012950    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-fv6z7"
Sep 08 13:26:07.135611 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:07.135540    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-9wk46"
Sep 08 13:26:07.285236 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:07.285208    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-fv6z7"]
Sep 08 13:26:07.287638 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:26:07.287607    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ce36be3_3649_478c_b536_af51dca1683d.slice/crio-a58832b80bbd3f85b13dcb592102c219c033b2db765ff40bef3d937e86826982 WatchSource:0}: Error finding container a58832b80bbd3f85b13dcb592102c219c033b2db765ff40bef3d937e86826982: Status 404 returned error can't find the container with id a58832b80bbd3f85b13dcb592102c219c033b2db765ff40bef3d937e86826982
Sep 08 13:26:07.404044 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:07.403853    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-9wk46"]
Sep 08 13:26:07.406535 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:26:07.406507    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode083282d_e401_4e6c_8a92_c3b1200f9341.slice/crio-1e540144530d9685a5e4b747af40f85612dfa369ea8af5240f5ad2b83473dd7a WatchSource:0}: Error finding container 1e540144530d9685a5e4b747af40f85612dfa369ea8af5240f5ad2b83473dd7a: Status 404 returned error can't find the container with id 1e540144530d9685a5e4b747af40f85612dfa369ea8af5240f5ad2b83473dd7a
Sep 08 13:26:07.639279 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:07.639238    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-fv6z7" event={"ID":"7ce36be3-3649-478c-b536-af51dca1683d","Type":"ContainerStarted","Data":"a58832b80bbd3f85b13dcb592102c219c033b2db765ff40bef3d937e86826982"}
Sep 08 13:26:07.640355 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:07.640322    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-9wk46" event={"ID":"e083282d-e401-4e6c-8a92-c3b1200f9341","Type":"ContainerStarted","Data":"1e540144530d9685a5e4b747af40f85612dfa369ea8af5240f5ad2b83473dd7a"}
Sep 08 13:26:12.668901 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:12.668860    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-fv6z7" event={"ID":"7ce36be3-3649-478c-b536-af51dca1683d","Type":"ContainerStarted","Data":"2402f1b330ed50248c2b77adeb860a9a04563723f412084ecebf69ecd7c5f653"}
Sep 08 13:26:12.670490 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:12.670461    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-9wk46" event={"ID":"e083282d-e401-4e6c-8a92-c3b1200f9341","Type":"ContainerStarted","Data":"b9bc6c5e7784634ece2d5c3de106c5cc663b06a4bc23bf6b689b523aa58a58cc"}
Sep 08 13:26:12.683642 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:12.683576    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-fv6z7" podStartSLOduration=1.611531102 podStartE2EDuration="6.683560295s" totalImagesPullingTime="5.072029193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:26:12.682483397 +0000 UTC m=+636.134244323" watchObservedRunningTime="2026-09-08 13:26:12.683560295 +0000 UTC m=+636.135321225"
Sep 08 13:26:12.695939 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:12.695892    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-9wk46" podStartSLOduration=1.731940367 podStartE2EDuration="6.695874964s" totalImagesPullingTime="4.963934597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:26:12.694909561 +0000 UTC m=+636.146670494" watchObservedRunningTime="2026-09-08 13:26:12.695874964 +0000 UTC m=+636.147635895"
Sep 08 13:26:12.722886 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:12.722846    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-fv6z7"]
Sep 08 13:26:14.679454 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:14.679411    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-fv6z7" podUID="7ce36be3-3649-478c-b536-af51dca1683d" containerName="authorino" containerID="cri-o://2402f1b330ed50248c2b77adeb860a9a04563723f412084ecebf69ecd7c5f653" gracePeriod=30
Sep 08 13:26:14.921994 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:14.921969    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-fv6z7"
Sep 08 13:26:14.997992 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:14.997904    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rqrd\" (UniqueName: \"kubernetes.io/projected/7ce36be3-3649-478c-b536-af51dca1683d-kube-api-access-4rqrd\") pod \"7ce36be3-3649-478c-b536-af51dca1683d\" (UID: \"7ce36be3-3649-478c-b536-af51dca1683d\") "
Sep 08 13:26:14.999957 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:14.999929    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ce36be3-3649-478c-b536-af51dca1683d-kube-api-access-4rqrd" (OuterVolumeSpecName: "kube-api-access-4rqrd") pod "7ce36be3-3649-478c-b536-af51dca1683d" (UID: "7ce36be3-3649-478c-b536-af51dca1683d"). InnerVolumeSpecName "kube-api-access-4rqrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:26:15.098863 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:15.098822    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rqrd\" (UniqueName: \"kubernetes.io/projected/7ce36be3-3649-478c-b536-af51dca1683d-kube-api-access-4rqrd\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:26:15.685302 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:15.685263    2558 generic.go:356] "Generic (PLEG): container finished" podID="7ce36be3-3649-478c-b536-af51dca1683d" containerID="2402f1b330ed50248c2b77adeb860a9a04563723f412084ecebf69ecd7c5f653" exitCode=0
Sep 08 13:26:15.685716 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:15.685306    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-fv6z7"
Sep 08 13:26:15.685716 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:15.685331    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-fv6z7" event={"ID":"7ce36be3-3649-478c-b536-af51dca1683d","Type":"ContainerDied","Data":"2402f1b330ed50248c2b77adeb860a9a04563723f412084ecebf69ecd7c5f653"}
Sep 08 13:26:15.685716 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:15.685369    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-fv6z7" event={"ID":"7ce36be3-3649-478c-b536-af51dca1683d","Type":"ContainerDied","Data":"a58832b80bbd3f85b13dcb592102c219c033b2db765ff40bef3d937e86826982"}
Sep 08 13:26:15.685716 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:15.685389    2558 scope.go:117] "RemoveContainer" containerID="2402f1b330ed50248c2b77adeb860a9a04563723f412084ecebf69ecd7c5f653"
Sep 08 13:26:15.693939 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:15.693915    2558 scope.go:117] "RemoveContainer" containerID="2402f1b330ed50248c2b77adeb860a9a04563723f412084ecebf69ecd7c5f653"
Sep 08 13:26:15.694244 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:26:15.694222    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2402f1b330ed50248c2b77adeb860a9a04563723f412084ecebf69ecd7c5f653\": container with ID starting with 2402f1b330ed50248c2b77adeb860a9a04563723f412084ecebf69ecd7c5f653 not found: ID does not exist" containerID="2402f1b330ed50248c2b77adeb860a9a04563723f412084ecebf69ecd7c5f653"
Sep 08 13:26:15.694314 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:15.694253    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2402f1b330ed50248c2b77adeb860a9a04563723f412084ecebf69ecd7c5f653"} err="failed to get container status \"2402f1b330ed50248c2b77adeb860a9a04563723f412084ecebf69ecd7c5f653\": rpc error: code = NotFound desc = could not find container \"2402f1b330ed50248c2b77adeb860a9a04563723f412084ecebf69ecd7c5f653\": container with ID starting with 2402f1b330ed50248c2b77adeb860a9a04563723f412084ecebf69ecd7c5f653 not found: ID does not exist"
Sep 08 13:26:15.700832 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:15.700810    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-fv6z7"]
Sep 08 13:26:15.706148 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:15.706124    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-fv6z7"]
Sep 08 13:26:17.057992 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:17.057960    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ce36be3-3649-478c-b536-af51dca1683d" path="/var/lib/kubelet/pods/7ce36be3-3649-478c-b536-af51dca1683d/volumes"
Sep 08 13:26:31.999523 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:31.999482    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7cf4765b85-66v2g"]
Sep 08 13:26:32.000014 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:31.999882    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ce36be3-3649-478c-b536-af51dca1683d" containerName="authorino"
Sep 08 13:26:32.000014 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:31.999895    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ce36be3-3649-478c-b536-af51dca1683d" containerName="authorino"
Sep 08 13:26:32.000014 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:31.999957    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ce36be3-3649-478c-b536-af51dca1683d" containerName="authorino"
Sep 08 13:26:32.002905 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:32.002888    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf4765b85-66v2g"
Sep 08 13:26:32.005076 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:32.005057    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 08 13:26:32.010192 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:32.010175    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cf4765b85-66v2g"]
Sep 08 13:26:32.142509 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:32.142472    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28p5d\" (UniqueName: \"kubernetes.io/projected/938161a3-9204-4b80-a381-eac9102eca91-kube-api-access-28p5d\") pod \"authorino-7cf4765b85-66v2g\" (UID: \"938161a3-9204-4b80-a381-eac9102eca91\") " pod="kuadrant-system/authorino-7cf4765b85-66v2g"
Sep 08 13:26:32.142720 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:32.142584    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/938161a3-9204-4b80-a381-eac9102eca91-tls-cert\") pod \"authorino-7cf4765b85-66v2g\" (UID: \"938161a3-9204-4b80-a381-eac9102eca91\") " pod="kuadrant-system/authorino-7cf4765b85-66v2g"
Sep 08 13:26:32.243188 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:32.243148    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/938161a3-9204-4b80-a381-eac9102eca91-tls-cert\") pod \"authorino-7cf4765b85-66v2g\" (UID: \"938161a3-9204-4b80-a381-eac9102eca91\") " pod="kuadrant-system/authorino-7cf4765b85-66v2g"
Sep 08 13:26:32.243357 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:32.243217    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28p5d\" (UniqueName: \"kubernetes.io/projected/938161a3-9204-4b80-a381-eac9102eca91-kube-api-access-28p5d\") pod \"authorino-7cf4765b85-66v2g\" (UID: \"938161a3-9204-4b80-a381-eac9102eca91\") " pod="kuadrant-system/authorino-7cf4765b85-66v2g"
Sep 08 13:26:32.245616 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:32.245574    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/938161a3-9204-4b80-a381-eac9102eca91-tls-cert\") pod \"authorino-7cf4765b85-66v2g\" (UID: \"938161a3-9204-4b80-a381-eac9102eca91\") " pod="kuadrant-system/authorino-7cf4765b85-66v2g"
Sep 08 13:26:32.252482 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:32.252419    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28p5d\" (UniqueName: \"kubernetes.io/projected/938161a3-9204-4b80-a381-eac9102eca91-kube-api-access-28p5d\") pod \"authorino-7cf4765b85-66v2g\" (UID: \"938161a3-9204-4b80-a381-eac9102eca91\") " pod="kuadrant-system/authorino-7cf4765b85-66v2g"
Sep 08 13:26:32.312955 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:32.312917    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf4765b85-66v2g"
Sep 08 13:26:32.583985 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:32.583906    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cf4765b85-66v2g"]
Sep 08 13:26:32.586909 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:26:32.586879    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod938161a3_9204_4b80_a381_eac9102eca91.slice/crio-648db45cd6898e86f0d7dfa3ae96388a0c2e6e9e81f057fadd0ae30ee19152c6 WatchSource:0}: Error finding container 648db45cd6898e86f0d7dfa3ae96388a0c2e6e9e81f057fadd0ae30ee19152c6: Status 404 returned error can't find the container with id 648db45cd6898e86f0d7dfa3ae96388a0c2e6e9e81f057fadd0ae30ee19152c6
Sep 08 13:26:32.763642 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:32.763577    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cf4765b85-66v2g" event={"ID":"938161a3-9204-4b80-a381-eac9102eca91","Type":"ContainerStarted","Data":"648db45cd6898e86f0d7dfa3ae96388a0c2e6e9e81f057fadd0ae30ee19152c6"}
Sep 08 13:26:33.770492 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:33.770455    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cf4765b85-66v2g" event={"ID":"938161a3-9204-4b80-a381-eac9102eca91","Type":"ContainerStarted","Data":"bddc17e9f8d7b2b1823f5c28ea7cb057a410b67623eee98b9f5feb0cc9f30304"}
Sep 08 13:26:33.784339 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:33.784287    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7cf4765b85-66v2g" podStartSLOduration=2.126127765 podStartE2EDuration="2.784270048s" totalImagesPullingTime="658.142283ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:26:33.783211376 +0000 UTC m=+657.234972304" watchObservedRunningTime="2026-09-08 13:26:33.784270048 +0000 UTC m=+657.236030979"
Sep 08 13:26:33.806883 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:33.806849    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-9wk46"]
Sep 08 13:26:33.807107 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:33.807082    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-9wk46" podUID="e083282d-e401-4e6c-8a92-c3b1200f9341" containerName="authorino" containerID="cri-o://b9bc6c5e7784634ece2d5c3de106c5cc663b06a4bc23bf6b689b523aa58a58cc" gracePeriod=30
Sep 08 13:26:34.048956 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:34.048930    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-9wk46"
Sep 08 13:26:34.157771 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:34.157726    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqwbn\" (UniqueName: \"kubernetes.io/projected/e083282d-e401-4e6c-8a92-c3b1200f9341-kube-api-access-hqwbn\") pod \"e083282d-e401-4e6c-8a92-c3b1200f9341\" (UID: \"e083282d-e401-4e6c-8a92-c3b1200f9341\") "
Sep 08 13:26:34.159931 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:34.159901    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e083282d-e401-4e6c-8a92-c3b1200f9341-kube-api-access-hqwbn" (OuterVolumeSpecName: "kube-api-access-hqwbn") pod "e083282d-e401-4e6c-8a92-c3b1200f9341" (UID: "e083282d-e401-4e6c-8a92-c3b1200f9341"). InnerVolumeSpecName "kube-api-access-hqwbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:26:34.259022 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:34.258992    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqwbn\" (UniqueName: \"kubernetes.io/projected/e083282d-e401-4e6c-8a92-c3b1200f9341-kube-api-access-hqwbn\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:26:34.775842 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:34.775799    2558 generic.go:356] "Generic (PLEG): container finished" podID="e083282d-e401-4e6c-8a92-c3b1200f9341" containerID="b9bc6c5e7784634ece2d5c3de106c5cc663b06a4bc23bf6b689b523aa58a58cc" exitCode=0
Sep 08 13:26:34.776269 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:34.775861    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-9wk46"
Sep 08 13:26:34.776269 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:34.775891    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-9wk46" event={"ID":"e083282d-e401-4e6c-8a92-c3b1200f9341","Type":"ContainerDied","Data":"b9bc6c5e7784634ece2d5c3de106c5cc663b06a4bc23bf6b689b523aa58a58cc"}
Sep 08 13:26:34.776269 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:34.775939    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-9wk46" event={"ID":"e083282d-e401-4e6c-8a92-c3b1200f9341","Type":"ContainerDied","Data":"1e540144530d9685a5e4b747af40f85612dfa369ea8af5240f5ad2b83473dd7a"}
Sep 08 13:26:34.776269 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:34.775958    2558 scope.go:117] "RemoveContainer" containerID="b9bc6c5e7784634ece2d5c3de106c5cc663b06a4bc23bf6b689b523aa58a58cc"
Sep 08 13:26:34.784982 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:34.784963    2558 scope.go:117] "RemoveContainer" containerID="b9bc6c5e7784634ece2d5c3de106c5cc663b06a4bc23bf6b689b523aa58a58cc"
Sep 08 13:26:34.785287 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:26:34.785270    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9bc6c5e7784634ece2d5c3de106c5cc663b06a4bc23bf6b689b523aa58a58cc\": container with ID starting with b9bc6c5e7784634ece2d5c3de106c5cc663b06a4bc23bf6b689b523aa58a58cc not found: ID does not exist" containerID="b9bc6c5e7784634ece2d5c3de106c5cc663b06a4bc23bf6b689b523aa58a58cc"
Sep 08 13:26:34.785333 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:34.785295    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9bc6c5e7784634ece2d5c3de106c5cc663b06a4bc23bf6b689b523aa58a58cc"} err="failed to get container status \"b9bc6c5e7784634ece2d5c3de106c5cc663b06a4bc23bf6b689b523aa58a58cc\": rpc error: code = NotFound desc = could not find container \"b9bc6c5e7784634ece2d5c3de106c5cc663b06a4bc23bf6b689b523aa58a58cc\": container with ID starting with b9bc6c5e7784634ece2d5c3de106c5cc663b06a4bc23bf6b689b523aa58a58cc not found: ID does not exist"
Sep 08 13:26:34.794548 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:34.794522    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-9wk46"]
Sep 08 13:26:34.797542 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:34.797506    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-9wk46"]
Sep 08 13:26:35.053804 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:26:35.053726    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e083282d-e401-4e6c-8a92-c3b1200f9341" path="/var/lib/kubelet/pods/e083282d-e401-4e6c-8a92-c3b1200f9341/volumes"
Sep 08 13:27:03.233301 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:27:03.233264    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-bql4r"]
Sep 08 13:27:03.233760 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:27:03.233646    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e083282d-e401-4e6c-8a92-c3b1200f9341" containerName="authorino"
Sep 08 13:27:03.233760 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:27:03.233660    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e083282d-e401-4e6c-8a92-c3b1200f9341" containerName="authorino"
Sep 08 13:27:03.233760 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:27:03.233718    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e083282d-e401-4e6c-8a92-c3b1200f9341" containerName="authorino"
Sep 08 13:27:03.236361 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:27:03.236342    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-bql4r"
Sep 08 13:27:03.238604 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:27:03.238572    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 08 13:27:03.239320 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:27:03.239299    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 08 13:27:03.239404 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:27:03.239341    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 08 13:27:03.239467 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:27:03.239413    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-h2h2x\""
Sep 08 13:27:03.244773 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:27:03.244746    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-bql4r"]
Sep 08 13:27:03.305968 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:27:03.305934    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/dfcdf8b7-003b-470f-bd21-19f5db5b0a4c-data\") pod \"seaweedfs-64568bcd49-bql4r\" (UID: \"dfcdf8b7-003b-470f-bd21-19f5db5b0a4c\") " pod="kserve/seaweedfs-64568bcd49-bql4r"
Sep 08 13:27:03.306154 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:27:03.306011    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwkpp\" (UniqueName: \"kubernetes.io/projected/dfcdf8b7-003b-470f-bd21-19f5db5b0a4c-kube-api-access-fwkpp\") pod \"seaweedfs-64568bcd49-bql4r\" (UID: \"dfcdf8b7-003b-470f-bd21-19f5db5b0a4c\") " pod="kserve/seaweedfs-64568bcd49-bql4r"
Sep 08 13:27:03.407408 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:27:03.407373    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwkpp\" (UniqueName: \"kubernetes.io/projected/dfcdf8b7-003b-470f-bd21-19f5db5b0a4c-kube-api-access-fwkpp\") pod \"seaweedfs-64568bcd49-bql4r\" (UID: \"dfcdf8b7-003b-470f-bd21-19f5db5b0a4c\") " pod="kserve/seaweedfs-64568bcd49-bql4r"
Sep 08 13:27:03.407554 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:27:03.407447    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/dfcdf8b7-003b-470f-bd21-19f5db5b0a4c-data\") pod \"seaweedfs-64568bcd49-bql4r\" (UID: \"dfcdf8b7-003b-470f-bd21-19f5db5b0a4c\") " pod="kserve/seaweedfs-64568bcd49-bql4r"
Sep 08 13:27:03.407870 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:27:03.407851    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/dfcdf8b7-003b-470f-bd21-19f5db5b0a4c-data\") pod \"seaweedfs-64568bcd49-bql4r\" (UID: \"dfcdf8b7-003b-470f-bd21-19f5db5b0a4c\") " pod="kserve/seaweedfs-64568bcd49-bql4r"
Sep 08 13:27:03.415404 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:27:03.415377    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwkpp\" (UniqueName: \"kubernetes.io/projected/dfcdf8b7-003b-470f-bd21-19f5db5b0a4c-kube-api-access-fwkpp\") pod \"seaweedfs-64568bcd49-bql4r\" (UID: \"dfcdf8b7-003b-470f-bd21-19f5db5b0a4c\") " pod="kserve/seaweedfs-64568bcd49-bql4r"
Sep 08 13:27:03.546243 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:27:03.546161    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-bql4r"
Sep 08 13:27:03.814565 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:27:03.814483    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-bql4r"]
Sep 08 13:27:03.818019 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:27:03.817992    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfcdf8b7_003b_470f_bd21_19f5db5b0a4c.slice/crio-fef46ad71fd98e2bf086682b5fd9e14974374c26ed3a0c613157f0f901786301 WatchSource:0}: Error finding container fef46ad71fd98e2bf086682b5fd9e14974374c26ed3a0c613157f0f901786301: Status 404 returned error can't find the container with id fef46ad71fd98e2bf086682b5fd9e14974374c26ed3a0c613157f0f901786301
Sep 08 13:27:03.907482 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:27:03.907446    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-bql4r" event={"ID":"dfcdf8b7-003b-470f-bd21-19f5db5b0a4c","Type":"ContainerStarted","Data":"fef46ad71fd98e2bf086682b5fd9e14974374c26ed3a0c613157f0f901786301"}
Sep 08 13:27:08.939178 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:27:08.939126    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-bql4r" event={"ID":"dfcdf8b7-003b-470f-bd21-19f5db5b0a4c","Type":"ContainerStarted","Data":"c551189e01154d95c9c3ef00c81c6313be8ff99a8bec28a4b63ab0fa79ddfeab"}
Sep 08 13:27:08.939635 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:27:08.939238    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-bql4r"
Sep 08 13:27:08.956002 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:27:08.955945    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-bql4r" podStartSLOduration=1.354084526 podStartE2EDuration="5.955928529s" totalImagesPullingTime="4.601844003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:27:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:27:08.955636077 +0000 UTC m=+692.407397011" watchObservedRunningTime="2026-09-08 13:27:08.955928529 +0000 UTC m=+692.407689496"
Sep 08 13:27:14.946909 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:27:14.946872    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-bql4r"
Sep 08 13:28:21.596655 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:21.596616    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-vcwmq"]
Sep 08 13:28:21.600182 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:21.600160    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-vcwmq"
Sep 08 13:28:21.603722 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:21.603694    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 08 13:28:21.603833 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:21.603760    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-nqngs\""
Sep 08 13:28:21.604265 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:21.604243    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 08 13:28:21.617998 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:21.617973    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-vcwmq"]
Sep 08 13:28:21.701706 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:21.701668    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/acb16799-7937-4c56-824b-fb8cc954cc22-cert\") pod \"odh-model-controller-7b966d7984-vcwmq\" (UID: \"acb16799-7937-4c56-824b-fb8cc954cc22\") " pod="kserve/odh-model-controller-7b966d7984-vcwmq"
Sep 08 13:28:21.701883 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:21.701720    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acb16799-7937-4c56-824b-fb8cc954cc22-metrics-certs\") pod \"odh-model-controller-7b966d7984-vcwmq\" (UID: \"acb16799-7937-4c56-824b-fb8cc954cc22\") " pod="kserve/odh-model-controller-7b966d7984-vcwmq"
Sep 08 13:28:21.701883 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:21.701766    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stwl9\" (UniqueName: \"kubernetes.io/projected/acb16799-7937-4c56-824b-fb8cc954cc22-kube-api-access-stwl9\") pod \"odh-model-controller-7b966d7984-vcwmq\" (UID: \"acb16799-7937-4c56-824b-fb8cc954cc22\") " pod="kserve/odh-model-controller-7b966d7984-vcwmq"
Sep 08 13:28:21.802675 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:21.802638    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/acb16799-7937-4c56-824b-fb8cc954cc22-cert\") pod \"odh-model-controller-7b966d7984-vcwmq\" (UID: \"acb16799-7937-4c56-824b-fb8cc954cc22\") " pod="kserve/odh-model-controller-7b966d7984-vcwmq"
Sep 08 13:28:21.802882 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:21.802710    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acb16799-7937-4c56-824b-fb8cc954cc22-metrics-certs\") pod \"odh-model-controller-7b966d7984-vcwmq\" (UID: \"acb16799-7937-4c56-824b-fb8cc954cc22\") " pod="kserve/odh-model-controller-7b966d7984-vcwmq"
Sep 08 13:28:21.802882 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:21.802757    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stwl9\" (UniqueName: \"kubernetes.io/projected/acb16799-7937-4c56-824b-fb8cc954cc22-kube-api-access-stwl9\") pod \"odh-model-controller-7b966d7984-vcwmq\" (UID: \"acb16799-7937-4c56-824b-fb8cc954cc22\") " pod="kserve/odh-model-controller-7b966d7984-vcwmq"
Sep 08 13:28:21.802882 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:28:21.802847    2558 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 08 13:28:21.803025 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:28:21.802920    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acb16799-7937-4c56-824b-fb8cc954cc22-metrics-certs podName:acb16799-7937-4c56-824b-fb8cc954cc22 nodeName:}" failed. No retries permitted until 2026-09-08 13:28:22.302904077 +0000 UTC m=+765.754664986 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/acb16799-7937-4c56-824b-fb8cc954cc22-metrics-certs") pod "odh-model-controller-7b966d7984-vcwmq" (UID: "acb16799-7937-4c56-824b-fb8cc954cc22") : secret "odh-model-controller-metrics-tls" not found
Sep 08 13:28:21.805312 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:21.805288    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/acb16799-7937-4c56-824b-fb8cc954cc22-cert\") pod \"odh-model-controller-7b966d7984-vcwmq\" (UID: \"acb16799-7937-4c56-824b-fb8cc954cc22\") " pod="kserve/odh-model-controller-7b966d7984-vcwmq"
Sep 08 13:28:21.812735 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:21.812711    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stwl9\" (UniqueName: \"kubernetes.io/projected/acb16799-7937-4c56-824b-fb8cc954cc22-kube-api-access-stwl9\") pod \"odh-model-controller-7b966d7984-vcwmq\" (UID: \"acb16799-7937-4c56-824b-fb8cc954cc22\") " pod="kserve/odh-model-controller-7b966d7984-vcwmq"
Sep 08 13:28:22.307473 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:22.307436    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acb16799-7937-4c56-824b-fb8cc954cc22-metrics-certs\") pod \"odh-model-controller-7b966d7984-vcwmq\" (UID: \"acb16799-7937-4c56-824b-fb8cc954cc22\") " pod="kserve/odh-model-controller-7b966d7984-vcwmq"
Sep 08 13:28:22.309852 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:22.309827    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acb16799-7937-4c56-824b-fb8cc954cc22-metrics-certs\") pod \"odh-model-controller-7b966d7984-vcwmq\" (UID: \"acb16799-7937-4c56-824b-fb8cc954cc22\") " pod="kserve/odh-model-controller-7b966d7984-vcwmq"
Sep 08 13:28:22.509915 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:22.509859    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-vcwmq"
Sep 08 13:28:22.786874 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:22.786688    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-vcwmq"]
Sep 08 13:28:22.789493 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:28:22.789466    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacb16799_7937_4c56_824b_fb8cc954cc22.slice/crio-b63f43bdd5b711503b94b3a5faa7201574bcd5fad831166caf57e341094e1c2c WatchSource:0}: Error finding container b63f43bdd5b711503b94b3a5faa7201574bcd5fad831166caf57e341094e1c2c: Status 404 returned error can't find the container with id b63f43bdd5b711503b94b3a5faa7201574bcd5fad831166caf57e341094e1c2c
Sep 08 13:28:22.790854 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:22.790836    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:28:23.276369 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:23.276334    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-vcwmq" event={"ID":"acb16799-7937-4c56-824b-fb8cc954cc22","Type":"ContainerStarted","Data":"b63f43bdd5b711503b94b3a5faa7201574bcd5fad831166caf57e341094e1c2c"}
Sep 08 13:28:26.295449 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:26.295408    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-vcwmq" event={"ID":"acb16799-7937-4c56-824b-fb8cc954cc22","Type":"ContainerStarted","Data":"f763911ec926090b3794fb5ef672f036d4a7718e31c7243e4badd4921b243d84"}
Sep 08 13:28:26.295874 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:26.295546    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-vcwmq"
Sep 08 13:28:26.317268 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:26.317214    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-vcwmq" podStartSLOduration=2.81110046 podStartE2EDuration="5.317199074s" totalImagesPullingTime="2.506098614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:28:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:28:26.315061926 +0000 UTC m=+769.766822858" watchObservedRunningTime="2026-09-08 13:28:26.317199074 +0000 UTC m=+769.768960005"
Sep 08 13:28:37.303338 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:37.303304    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-vcwmq"
Sep 08 13:28:38.152335 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:38.152298    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-dnq2z"]
Sep 08 13:28:38.157363 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:38.157338    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-dnq2z"
Sep 08 13:28:38.160098 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:38.160078    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 08 13:28:38.167726 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:38.167704    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-dnq2z"]
Sep 08 13:28:38.242436 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:38.242404    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slljf\" (UniqueName: \"kubernetes.io/projected/ef854386-230c-47e6-9e11-e538b12a9e68-kube-api-access-slljf\") pod \"s3-init-dnq2z\" (UID: \"ef854386-230c-47e6-9e11-e538b12a9e68\") " pod="kserve/s3-init-dnq2z"
Sep 08 13:28:38.242616 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:38.242573    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef854386-230c-47e6-9e11-e538b12a9e68-model-cache\") pod \"s3-init-dnq2z\" (UID: \"ef854386-230c-47e6-9e11-e538b12a9e68\") " pod="kserve/s3-init-dnq2z"
Sep 08 13:28:38.343817 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:38.343763    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef854386-230c-47e6-9e11-e538b12a9e68-model-cache\") pod \"s3-init-dnq2z\" (UID: \"ef854386-230c-47e6-9e11-e538b12a9e68\") " pod="kserve/s3-init-dnq2z"
Sep 08 13:28:38.343817 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:38.343827    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slljf\" (UniqueName: \"kubernetes.io/projected/ef854386-230c-47e6-9e11-e538b12a9e68-kube-api-access-slljf\") pod \"s3-init-dnq2z\" (UID: \"ef854386-230c-47e6-9e11-e538b12a9e68\") " pod="kserve/s3-init-dnq2z"
Sep 08 13:28:38.344266 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:38.344241    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef854386-230c-47e6-9e11-e538b12a9e68-model-cache\") pod \"s3-init-dnq2z\" (UID: \"ef854386-230c-47e6-9e11-e538b12a9e68\") " pod="kserve/s3-init-dnq2z"
Sep 08 13:28:38.353377 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:38.353351    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slljf\" (UniqueName: \"kubernetes.io/projected/ef854386-230c-47e6-9e11-e538b12a9e68-kube-api-access-slljf\") pod \"s3-init-dnq2z\" (UID: \"ef854386-230c-47e6-9e11-e538b12a9e68\") " pod="kserve/s3-init-dnq2z"
Sep 08 13:28:38.467659 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:38.467573    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-dnq2z"
Sep 08 13:28:38.743867 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:38.743843    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-dnq2z"]
Sep 08 13:28:38.746151 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:28:38.746118    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef854386_230c_47e6_9e11_e538b12a9e68.slice/crio-21fe34ca81a3daebc2ee7e7be3903c2b0551aa0ec2fdd517c0cdd76ec84d7945 WatchSource:0}: Error finding container 21fe34ca81a3daebc2ee7e7be3903c2b0551aa0ec2fdd517c0cdd76ec84d7945: Status 404 returned error can't find the container with id 21fe34ca81a3daebc2ee7e7be3903c2b0551aa0ec2fdd517c0cdd76ec84d7945
Sep 08 13:28:39.357111 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:39.357073    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-dnq2z" event={"ID":"ef854386-230c-47e6-9e11-e538b12a9e68","Type":"ContainerStarted","Data":"21fe34ca81a3daebc2ee7e7be3903c2b0551aa0ec2fdd517c0cdd76ec84d7945"}
Sep 08 13:28:42.375723 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:42.375682    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-dnq2z" event={"ID":"ef854386-230c-47e6-9e11-e538b12a9e68","Type":"ContainerStarted","Data":"f666fa28e2e2f332ea55ef35235641d7f4075ffce31cfe400e03927f5a384483"}
Sep 08 13:28:44.386398 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:44.386364    2558 generic.go:356] "Generic (PLEG): container finished" podID="ef854386-230c-47e6-9e11-e538b12a9e68" containerID="f666fa28e2e2f332ea55ef35235641d7f4075ffce31cfe400e03927f5a384483" exitCode=0
Sep 08 13:28:44.386783 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:44.386432    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-dnq2z" event={"ID":"ef854386-230c-47e6-9e11-e538b12a9e68","Type":"ContainerDied","Data":"f666fa28e2e2f332ea55ef35235641d7f4075ffce31cfe400e03927f5a384483"}
Sep 08 13:28:49.416503 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:49.416469    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-dnq2z" event={"ID":"ef854386-230c-47e6-9e11-e538b12a9e68","Type":"ContainerStarted","Data":"72d9504fd95b636d130b5313507e2b5e3ee55913a1642bc03d1ee6ad3deb254d"}
Sep 08 13:28:49.446621 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:28:49.446545    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-dnq2z" podStartSLOduration=2.05692211 podStartE2EDuration="11.446528226s" totalImagesPullingTime="7.782193624s" totalInitContainerRuntime="1.607412492s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:28:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:28:49.443378259 +0000 UTC m=+792.895139190" watchObservedRunningTime="2026-09-08 13:28:49.446528226 +0000 UTC m=+792.898289157"
Sep 08 13:29:54.731104 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:29:54.731071    2558 generic.go:356] "Generic (PLEG): container finished" podID="ef854386-230c-47e6-9e11-e538b12a9e68" containerID="72d9504fd95b636d130b5313507e2b5e3ee55913a1642bc03d1ee6ad3deb254d" exitCode=0
Sep 08 13:29:54.731516 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:29:54.731148    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-dnq2z" event={"ID":"ef854386-230c-47e6-9e11-e538b12a9e68","Type":"ContainerDied","Data":"72d9504fd95b636d130b5313507e2b5e3ee55913a1642bc03d1ee6ad3deb254d"}
Sep 08 13:29:55.868021 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:29:55.867993    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-dnq2z"
Sep 08 13:29:56.015967 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:29:56.015876    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef854386-230c-47e6-9e11-e538b12a9e68-model-cache\") pod \"ef854386-230c-47e6-9e11-e538b12a9e68\" (UID: \"ef854386-230c-47e6-9e11-e538b12a9e68\") "
Sep 08 13:29:56.015967 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:29:56.015931    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slljf\" (UniqueName: \"kubernetes.io/projected/ef854386-230c-47e6-9e11-e538b12a9e68-kube-api-access-slljf\") pod \"ef854386-230c-47e6-9e11-e538b12a9e68\" (UID: \"ef854386-230c-47e6-9e11-e538b12a9e68\") "
Sep 08 13:29:56.018135 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:29:56.018105    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef854386-230c-47e6-9e11-e538b12a9e68-kube-api-access-slljf" (OuterVolumeSpecName: "kube-api-access-slljf") pod "ef854386-230c-47e6-9e11-e538b12a9e68" (UID: "ef854386-230c-47e6-9e11-e538b12a9e68"). InnerVolumeSpecName "kube-api-access-slljf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:29:56.034623 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:29:56.034576    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef854386-230c-47e6-9e11-e538b12a9e68-model-cache" (OuterVolumeSpecName: "model-cache") pod "ef854386-230c-47e6-9e11-e538b12a9e68" (UID: "ef854386-230c-47e6-9e11-e538b12a9e68"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:29:56.117272 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:29:56.117232    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef854386-230c-47e6-9e11-e538b12a9e68-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:29:56.117272 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:29:56.117273    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slljf\" (UniqueName: \"kubernetes.io/projected/ef854386-230c-47e6-9e11-e538b12a9e68-kube-api-access-slljf\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:29:56.742931 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:29:56.742900    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-dnq2z"
Sep 08 13:29:56.743101 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:29:56.742928    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-dnq2z" event={"ID":"ef854386-230c-47e6-9e11-e538b12a9e68","Type":"ContainerDied","Data":"21fe34ca81a3daebc2ee7e7be3903c2b0551aa0ec2fdd517c0cdd76ec84d7945"}
Sep 08 13:29:56.743101 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:29:56.742966    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21fe34ca81a3daebc2ee7e7be3903c2b0551aa0ec2fdd517c0cdd76ec84d7945"
Sep 08 13:30:17.535683 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.535640    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"]
Sep 08 13:30:17.536102 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.536044    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef854386-230c-47e6-9e11-e538b12a9e68" containerName="download-hf-model"
Sep 08 13:30:17.536102 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.536057    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef854386-230c-47e6-9e11-e538b12a9e68" containerName="download-hf-model"
Sep 08 13:30:17.536102 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.536074    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef854386-230c-47e6-9e11-e538b12a9e68" containerName="s3-init"
Sep 08 13:30:17.536102 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.536081    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef854386-230c-47e6-9e11-e538b12a9e68" containerName="s3-init"
Sep 08 13:30:17.536230 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.536140    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef854386-230c-47e6-9e11-e538b12a9e68" containerName="s3-init"
Sep 08 13:30:17.539202 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.539185    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"
Sep 08 13:30:17.547419 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.547388    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Sep 08 13:30:17.547419 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.547388    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-qd2ml\""
Sep 08 13:30:17.547645 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.547439    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Sep 08 13:30:17.547645 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.547502    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:30:17.547645 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.547391    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Sep 08 13:30:17.547854 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.547832    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Sep 08 13:30:17.603540 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.603500    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/930a0152-d53c-4929-bd91-8d9267c164d4-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg\" (UID: \"930a0152-d53c-4929-bd91-8d9267c164d4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"
Sep 08 13:30:17.603540 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.603541    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/930a0152-d53c-4929-bd91-8d9267c164d4-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg\" (UID: \"930a0152-d53c-4929-bd91-8d9267c164d4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"
Sep 08 13:30:17.603826 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.603581    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/930a0152-d53c-4929-bd91-8d9267c164d4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg\" (UID: \"930a0152-d53c-4929-bd91-8d9267c164d4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"
Sep 08 13:30:17.603826 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.603699    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/930a0152-d53c-4929-bd91-8d9267c164d4-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg\" (UID: \"930a0152-d53c-4929-bd91-8d9267c164d4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"
Sep 08 13:30:17.603826 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.603755    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930a0152-d53c-4929-bd91-8d9267c164d4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg\" (UID: \"930a0152-d53c-4929-bd91-8d9267c164d4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"
Sep 08 13:30:17.603980 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.603829    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/930a0152-d53c-4929-bd91-8d9267c164d4-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg\" (UID: \"930a0152-d53c-4929-bd91-8d9267c164d4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"
Sep 08 13:30:17.603980 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.603863    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/930a0152-d53c-4929-bd91-8d9267c164d4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg\" (UID: \"930a0152-d53c-4929-bd91-8d9267c164d4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"
Sep 08 13:30:17.603980 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.603897    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjvsb\" (UniqueName: \"kubernetes.io/projected/930a0152-d53c-4929-bd91-8d9267c164d4-kube-api-access-xjvsb\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg\" (UID: \"930a0152-d53c-4929-bd91-8d9267c164d4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"
Sep 08 13:30:17.619825 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.619789    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"]
Sep 08 13:30:17.704783 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.704746    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/930a0152-d53c-4929-bd91-8d9267c164d4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg\" (UID: \"930a0152-d53c-4929-bd91-8d9267c164d4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"
Sep 08 13:30:17.704976 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.704794    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/930a0152-d53c-4929-bd91-8d9267c164d4-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg\" (UID: \"930a0152-d53c-4929-bd91-8d9267c164d4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"
Sep 08 13:30:17.705042 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.704966    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930a0152-d53c-4929-bd91-8d9267c164d4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg\" (UID: \"930a0152-d53c-4929-bd91-8d9267c164d4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"
Sep 08 13:30:17.705042 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.705037    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/930a0152-d53c-4929-bd91-8d9267c164d4-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg\" (UID: \"930a0152-d53c-4929-bd91-8d9267c164d4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"
Sep 08 13:30:17.705154 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.705066    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/930a0152-d53c-4929-bd91-8d9267c164d4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg\" (UID: \"930a0152-d53c-4929-bd91-8d9267c164d4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"
Sep 08 13:30:17.705154 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.705090    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjvsb\" (UniqueName: \"kubernetes.io/projected/930a0152-d53c-4929-bd91-8d9267c164d4-kube-api-access-xjvsb\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg\" (UID: \"930a0152-d53c-4929-bd91-8d9267c164d4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"
Sep 08 13:30:17.705154 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.705121    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/930a0152-d53c-4929-bd91-8d9267c164d4-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg\" (UID: \"930a0152-d53c-4929-bd91-8d9267c164d4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"
Sep 08 13:30:17.705298 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.705153    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/930a0152-d53c-4929-bd91-8d9267c164d4-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg\" (UID: \"930a0152-d53c-4929-bd91-8d9267c164d4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"
Sep 08 13:30:17.705298 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.705206    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/930a0152-d53c-4929-bd91-8d9267c164d4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg\" (UID: \"930a0152-d53c-4929-bd91-8d9267c164d4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"
Sep 08 13:30:17.705298 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.705156    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/930a0152-d53c-4929-bd91-8d9267c164d4-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg\" (UID: \"930a0152-d53c-4929-bd91-8d9267c164d4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"
Sep 08 13:30:17.705461 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.705310    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930a0152-d53c-4929-bd91-8d9267c164d4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg\" (UID: \"930a0152-d53c-4929-bd91-8d9267c164d4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"
Sep 08 13:30:17.705521 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.705475    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/930a0152-d53c-4929-bd91-8d9267c164d4-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg\" (UID: \"930a0152-d53c-4929-bd91-8d9267c164d4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"
Sep 08 13:30:17.705856 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.705832    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/930a0152-d53c-4929-bd91-8d9267c164d4-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg\" (UID: \"930a0152-d53c-4929-bd91-8d9267c164d4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"
Sep 08 13:30:17.707255 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.707236    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/930a0152-d53c-4929-bd91-8d9267c164d4-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg\" (UID: \"930a0152-d53c-4929-bd91-8d9267c164d4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"
Sep 08 13:30:17.707515 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.707497    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/930a0152-d53c-4929-bd91-8d9267c164d4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg\" (UID: \"930a0152-d53c-4929-bd91-8d9267c164d4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"
Sep 08 13:30:17.713899 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.713861    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjvsb\" (UniqueName: \"kubernetes.io/projected/930a0152-d53c-4929-bd91-8d9267c164d4-kube-api-access-xjvsb\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg\" (UID: \"930a0152-d53c-4929-bd91-8d9267c164d4\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"
Sep 08 13:30:17.849543 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:17.849436    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"
Sep 08 13:30:18.143446 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:18.143402    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"]
Sep 08 13:30:18.147935 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:30:18.147869    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod930a0152_d53c_4929_bd91_8d9267c164d4.slice/crio-0134ba344af86c851179e21ec837aef1355e3c790fb3ac959ad725287a03dfc9 WatchSource:0}: Error finding container 0134ba344af86c851179e21ec837aef1355e3c790fb3ac959ad725287a03dfc9: Status 404 returned error can't find the container with id 0134ba344af86c851179e21ec837aef1355e3c790fb3ac959ad725287a03dfc9
Sep 08 13:30:18.850787 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:18.850737    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg" event={"ID":"930a0152-d53c-4929-bd91-8d9267c164d4","Type":"ContainerStarted","Data":"96208b444a866cc3041915368815180fcee2e33cd1a1c67bbd78f971f3320aa6"}
Sep 08 13:30:18.850787 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:18.850796    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg" event={"ID":"930a0152-d53c-4929-bd91-8d9267c164d4","Type":"ContainerStarted","Data":"0134ba344af86c851179e21ec837aef1355e3c790fb3ac959ad725287a03dfc9"}
Sep 08 13:30:20.863427 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:20.863390    2558 generic.go:356] "Generic (PLEG): container finished" podID="930a0152-d53c-4929-bd91-8d9267c164d4" containerID="96208b444a866cc3041915368815180fcee2e33cd1a1c67bbd78f971f3320aa6" exitCode=0
Sep 08 13:30:20.863922 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:20.863453    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg" event={"ID":"930a0152-d53c-4929-bd91-8d9267c164d4","Type":"ContainerDied","Data":"96208b444a866cc3041915368815180fcee2e33cd1a1c67bbd78f971f3320aa6"}
Sep 08 13:30:50.053852 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:50.053816    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg" event={"ID":"930a0152-d53c-4929-bd91-8d9267c164d4","Type":"ContainerStarted","Data":"823b5dce07454274e910266f53094e045c0b8dccac48a9a977528acacc47d5c9"}
Sep 08 13:30:50.077673 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:50.077608    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg" podStartSLOduration=2.130081947 podStartE2EDuration="33.077577281s" totalImagesPullingTime="28.559688314s" totalInitContainerRuntime="2.38780702s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:30:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:30:50.076072085 +0000 UTC m=+913.527833013" watchObservedRunningTime="2026-09-08 13:30:50.077577281 +0000 UTC m=+913.529338238"
Sep 08 13:30:50.850376 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:50.850340    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"
Sep 08 13:30:57.850082 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:57.850038    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"
Sep 08 13:30:57.851765 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:30:57.851734    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg" podUID="930a0152-d53c-4929-bd91-8d9267c164d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 08 13:31:07.850577 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:31:07.850532    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg" podUID="930a0152-d53c-4929-bd91-8d9267c164d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 08 13:31:17.850172 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:31:17.850118    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg" podUID="930a0152-d53c-4929-bd91-8d9267c164d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 08 13:31:27.850599 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:31:27.850540    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg" podUID="930a0152-d53c-4929-bd91-8d9267c164d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 08 13:31:37.850539 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:31:37.850497    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg" podUID="930a0152-d53c-4929-bd91-8d9267c164d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 08 13:31:47.850799 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:31:47.850751    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg" podUID="930a0152-d53c-4929-bd91-8d9267c164d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 08 13:31:57.850404 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:31:57.850358    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg" podUID="930a0152-d53c-4929-bd91-8d9267c164d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 08 13:32:07.850085 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:32:07.849990    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg" podUID="930a0152-d53c-4929-bd91-8d9267c164d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 08 13:32:17.850335 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:32:17.850288    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg" podUID="930a0152-d53c-4929-bd91-8d9267c164d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 08 13:32:27.849979 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:32:27.849932    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg" podUID="930a0152-d53c-4929-bd91-8d9267c164d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 08 13:32:37.850223 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:32:37.850168    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg" podUID="930a0152-d53c-4929-bd91-8d9267c164d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 08 13:32:47.860364 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:32:47.860324    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"
Sep 08 13:32:47.868360 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:32:47.868334    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"
Sep 08 13:32:52.449262 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:32:52.449229    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"]
Sep 08 13:33:07.472076 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:07.472013    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg" podUID="930a0152-d53c-4929-bd91-8d9267c164d4" containerName="main" containerID="cri-o://823b5dce07454274e910266f53094e045c0b8dccac48a9a977528acacc47d5c9" gracePeriod=45
Sep 08 13:33:07.861341 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:07.861301    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg" podUID="930a0152-d53c-4929-bd91-8d9267c164d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 08 13:33:08.221035 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.221008    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"
Sep 08 13:33:08.352869 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.352816    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjvsb\" (UniqueName: \"kubernetes.io/projected/930a0152-d53c-4929-bd91-8d9267c164d4-kube-api-access-xjvsb\") pod \"930a0152-d53c-4929-bd91-8d9267c164d4\" (UID: \"930a0152-d53c-4929-bd91-8d9267c164d4\") "
Sep 08 13:33:08.353078 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.352895    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930a0152-d53c-4929-bd91-8d9267c164d4-tmp-dir\") pod \"930a0152-d53c-4929-bd91-8d9267c164d4\" (UID: \"930a0152-d53c-4929-bd91-8d9267c164d4\") "
Sep 08 13:33:08.353078 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.352921    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/930a0152-d53c-4929-bd91-8d9267c164d4-tls-certs\") pod \"930a0152-d53c-4929-bd91-8d9267c164d4\" (UID: \"930a0152-d53c-4929-bd91-8d9267c164d4\") "
Sep 08 13:33:08.353078 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.352960    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/930a0152-d53c-4929-bd91-8d9267c164d4-model-cache\") pod \"930a0152-d53c-4929-bd91-8d9267c164d4\" (UID: \"930a0152-d53c-4929-bd91-8d9267c164d4\") "
Sep 08 13:33:08.353078 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.352991    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/930a0152-d53c-4929-bd91-8d9267c164d4-home\") pod \"930a0152-d53c-4929-bd91-8d9267c164d4\" (UID: \"930a0152-d53c-4929-bd91-8d9267c164d4\") "
Sep 08 13:33:08.353078 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.353016    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/930a0152-d53c-4929-bd91-8d9267c164d4-cabundle-cert\") pod \"930a0152-d53c-4929-bd91-8d9267c164d4\" (UID: \"930a0152-d53c-4929-bd91-8d9267c164d4\") "
Sep 08 13:33:08.353332 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.353087    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/930a0152-d53c-4929-bd91-8d9267c164d4-kserve-provision-location\") pod \"930a0152-d53c-4929-bd91-8d9267c164d4\" (UID: \"930a0152-d53c-4929-bd91-8d9267c164d4\") "
Sep 08 13:33:08.353332 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.353125    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/930a0152-d53c-4929-bd91-8d9267c164d4-dshm\") pod \"930a0152-d53c-4929-bd91-8d9267c164d4\" (UID: \"930a0152-d53c-4929-bd91-8d9267c164d4\") "
Sep 08 13:33:08.353332 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.353261    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/930a0152-d53c-4929-bd91-8d9267c164d4-model-cache" (OuterVolumeSpecName: "model-cache") pod "930a0152-d53c-4929-bd91-8d9267c164d4" (UID: "930a0152-d53c-4929-bd91-8d9267c164d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:33:08.353491 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.353447    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/930a0152-d53c-4929-bd91-8d9267c164d4-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:33:08.353653 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.353587    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/930a0152-d53c-4929-bd91-8d9267c164d4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "930a0152-d53c-4929-bd91-8d9267c164d4" (UID: "930a0152-d53c-4929-bd91-8d9267c164d4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:33:08.353788 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.353739    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/930a0152-d53c-4929-bd91-8d9267c164d4-home" (OuterVolumeSpecName: "home") pod "930a0152-d53c-4929-bd91-8d9267c164d4" (UID: "930a0152-d53c-4929-bd91-8d9267c164d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:33:08.355389 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.355362    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/930a0152-d53c-4929-bd91-8d9267c164d4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "930a0152-d53c-4929-bd91-8d9267c164d4" (UID: "930a0152-d53c-4929-bd91-8d9267c164d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:33:08.355773 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.355754    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/930a0152-d53c-4929-bd91-8d9267c164d4-kube-api-access-xjvsb" (OuterVolumeSpecName: "kube-api-access-xjvsb") pod "930a0152-d53c-4929-bd91-8d9267c164d4" (UID: "930a0152-d53c-4929-bd91-8d9267c164d4"). InnerVolumeSpecName "kube-api-access-xjvsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:33:08.355857 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.355794    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/930a0152-d53c-4929-bd91-8d9267c164d4-dshm" (OuterVolumeSpecName: "dshm") pod "930a0152-d53c-4929-bd91-8d9267c164d4" (UID: "930a0152-d53c-4929-bd91-8d9267c164d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:33:08.371339 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.371291    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/930a0152-d53c-4929-bd91-8d9267c164d4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "930a0152-d53c-4929-bd91-8d9267c164d4" (UID: "930a0152-d53c-4929-bd91-8d9267c164d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:33:08.373995 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.373964    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/930a0152-d53c-4929-bd91-8d9267c164d4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "930a0152-d53c-4929-bd91-8d9267c164d4" (UID: "930a0152-d53c-4929-bd91-8d9267c164d4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:33:08.454664 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.454534    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/930a0152-d53c-4929-bd91-8d9267c164d4-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:33:08.454664 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.454570    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjvsb\" (UniqueName: \"kubernetes.io/projected/930a0152-d53c-4929-bd91-8d9267c164d4-kube-api-access-xjvsb\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:33:08.454664 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.454584    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930a0152-d53c-4929-bd91-8d9267c164d4-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:33:08.454664 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.454622    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/930a0152-d53c-4929-bd91-8d9267c164d4-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:33:08.454664 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.454635    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/930a0152-d53c-4929-bd91-8d9267c164d4-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:33:08.454664 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.454646    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/930a0152-d53c-4929-bd91-8d9267c164d4-cabundle-cert\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:33:08.454664 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.454658    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/930a0152-d53c-4929-bd91-8d9267c164d4-kserve-provision-location\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:33:08.751666 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.751548    2558 generic.go:356] "Generic (PLEG): container finished" podID="930a0152-d53c-4929-bd91-8d9267c164d4" containerID="823b5dce07454274e910266f53094e045c0b8dccac48a9a977528acacc47d5c9" exitCode=0
Sep 08 13:33:08.751666 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.751637    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"
Sep 08 13:33:08.751666 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.751643    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg" event={"ID":"930a0152-d53c-4929-bd91-8d9267c164d4","Type":"ContainerDied","Data":"823b5dce07454274e910266f53094e045c0b8dccac48a9a977528acacc47d5c9"}
Sep 08 13:33:08.752238 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.751698    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg" event={"ID":"930a0152-d53c-4929-bd91-8d9267c164d4","Type":"ContainerDied","Data":"0134ba344af86c851179e21ec837aef1355e3c790fb3ac959ad725287a03dfc9"}
Sep 08 13:33:08.752238 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.751720    2558 scope.go:117] "RemoveContainer" containerID="823b5dce07454274e910266f53094e045c0b8dccac48a9a977528acacc47d5c9"
Sep 08 13:33:08.761279 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.761256    2558 scope.go:117] "RemoveContainer" containerID="96208b444a866cc3041915368815180fcee2e33cd1a1c67bbd78f971f3320aa6"
Sep 08 13:33:08.769329 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.769310    2558 scope.go:117] "RemoveContainer" containerID="823b5dce07454274e910266f53094e045c0b8dccac48a9a977528acacc47d5c9"
Sep 08 13:33:08.769632 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:33:08.769612    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"823b5dce07454274e910266f53094e045c0b8dccac48a9a977528acacc47d5c9\": container with ID starting with 823b5dce07454274e910266f53094e045c0b8dccac48a9a977528acacc47d5c9 not found: ID does not exist" containerID="823b5dce07454274e910266f53094e045c0b8dccac48a9a977528acacc47d5c9"
Sep 08 13:33:08.769688 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.769642    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"823b5dce07454274e910266f53094e045c0b8dccac48a9a977528acacc47d5c9"} err="failed to get container status \"823b5dce07454274e910266f53094e045c0b8dccac48a9a977528acacc47d5c9\": rpc error: code = NotFound desc = could not find container \"823b5dce07454274e910266f53094e045c0b8dccac48a9a977528acacc47d5c9\": container with ID starting with 823b5dce07454274e910266f53094e045c0b8dccac48a9a977528acacc47d5c9 not found: ID does not exist"
Sep 08 13:33:08.769688 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.769662    2558 scope.go:117] "RemoveContainer" containerID="96208b444a866cc3041915368815180fcee2e33cd1a1c67bbd78f971f3320aa6"
Sep 08 13:33:08.769964 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:33:08.769945    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96208b444a866cc3041915368815180fcee2e33cd1a1c67bbd78f971f3320aa6\": container with ID starting with 96208b444a866cc3041915368815180fcee2e33cd1a1c67bbd78f971f3320aa6 not found: ID does not exist" containerID="96208b444a866cc3041915368815180fcee2e33cd1a1c67bbd78f971f3320aa6"
Sep 08 13:33:08.770029 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.769972    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96208b444a866cc3041915368815180fcee2e33cd1a1c67bbd78f971f3320aa6"} err="failed to get container status \"96208b444a866cc3041915368815180fcee2e33cd1a1c67bbd78f971f3320aa6\": rpc error: code = NotFound desc = could not find container \"96208b444a866cc3041915368815180fcee2e33cd1a1c67bbd78f971f3320aa6\": container with ID starting with 96208b444a866cc3041915368815180fcee2e33cd1a1c67bbd78f971f3320aa6 not found: ID does not exist"
Sep 08 13:33:08.773247 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.773226    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"]
Sep 08 13:33:08.778008 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:08.777973    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-75cc47dc8bjh4fg"]
Sep 08 13:33:09.053127 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:09.053042    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="930a0152-d53c-4929-bd91-8d9267c164d4" path="/var/lib/kubelet/pods/930a0152-d53c-4929-bd91-8d9267c164d4/volumes"
Sep 08 13:33:34.860865 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:34.860790    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc"]
Sep 08 13:33:34.861218 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:34.861163    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="930a0152-d53c-4929-bd91-8d9267c164d4" containerName="storage-initializer"
Sep 08 13:33:34.861218 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:34.861175    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="930a0152-d53c-4929-bd91-8d9267c164d4" containerName="storage-initializer"
Sep 08 13:33:34.861218 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:34.861189    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="930a0152-d53c-4929-bd91-8d9267c164d4" containerName="main"
Sep 08 13:33:34.861218 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:34.861195    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="930a0152-d53c-4929-bd91-8d9267c164d4" containerName="main"
Sep 08 13:33:34.861349 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:34.861262    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="930a0152-d53c-4929-bd91-8d9267c164d4" containerName="main"
Sep 08 13:33:34.864345 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:34.864324    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc"
Sep 08 13:33:34.867171 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:34.867148    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-jtlxb\""
Sep 08 13:33:34.867773 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:34.867751    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 08 13:33:34.867869 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:34.867855    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 08 13:33:34.867911 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:34.867891    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-bxwrt\""
Sep 08 13:33:34.873963 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:34.873943    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 08 13:33:34.883306 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:34.883283    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc"]
Sep 08 13:33:34.897526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:34.897498    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 08 13:33:34.901058 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:34.901033    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 08 13:33:34.904137 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:34.904115    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:33:34.904243 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:34.904169    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 08 13:33:34.904243 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:34.904179    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-xwjms\""
Sep 08 13:33:34.944898 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:34.944868    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 08 13:33:34.979772 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:34.979739    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8kdz\" (UniqueName: \"kubernetes.io/projected/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-kube-api-access-r8kdz\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 08 13:33:34.979931 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:34.979783    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 08 13:33:34.979931 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:34.979809    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 08 13:33:34.979931 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:34.979889    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 08 13:33:34.979931 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:34.979913    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 08 13:33:34.980100 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:34.979943    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 08 13:33:34.980100 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:34.980030    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 08 13:33:34.980100 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:34.980067    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc2j9\" (UniqueName: \"kubernetes.io/projected/cfc4a75e-814d-4ea2-9581-763b03d73542-kube-api-access-dc2j9\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc\" (UID: \"cfc4a75e-814d-4ea2-9581-763b03d73542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc"
Sep 08 13:33:34.980100 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:34.980097    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc4a75e-814d-4ea2-9581-763b03d73542-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc\" (UID: \"cfc4a75e-814d-4ea2-9581-763b03d73542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc"
Sep 08 13:33:34.980224 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:34.980114    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 08 13:33:35.080763 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:35.080725    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 08 13:33:35.080945 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:35.080781    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc2j9\" (UniqueName: \"kubernetes.io/projected/cfc4a75e-814d-4ea2-9581-763b03d73542-kube-api-access-dc2j9\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc\" (UID: \"cfc4a75e-814d-4ea2-9581-763b03d73542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc"
Sep 08 13:33:35.080945 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:35.080827    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc4a75e-814d-4ea2-9581-763b03d73542-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc\" (UID: \"cfc4a75e-814d-4ea2-9581-763b03d73542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc"
Sep 08 13:33:35.080945 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:35.080844    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 08 13:33:35.080945 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:35.080877    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8kdz\" (UniqueName: \"kubernetes.io/projected/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-kube-api-access-r8kdz\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 08 13:33:35.080945 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:35.080912    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 08 13:33:35.081217 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:35.080950    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 08 13:33:35.081217 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:35.081014    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 08 13:33:35.081217 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:35.081057    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 08 13:33:35.081217 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:35.081079    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 08 13:33:35.081217 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:35.081190    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 08 13:33:35.081474 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:35.081384    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 08 13:33:35.081474 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:35.081450    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 08 13:33:35.081693 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:35.081667    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 08 13:33:35.081889 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:35.081866    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 08 13:33:35.083634 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:35.083612    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 08 13:33:35.083722 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:35.083645    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 08 13:33:35.083893 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:35.083877    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc4a75e-814d-4ea2-9581-763b03d73542-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc\" (UID: \"cfc4a75e-814d-4ea2-9581-763b03d73542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc"
Sep 08 13:33:35.090900 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:35.090874    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8kdz\" (UniqueName: \"kubernetes.io/projected/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-kube-api-access-r8kdz\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 08 13:33:35.091006 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:35.090939    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc2j9\" (UniqueName: \"kubernetes.io/projected/cfc4a75e-814d-4ea2-9581-763b03d73542-kube-api-access-dc2j9\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc\" (UID: \"cfc4a75e-814d-4ea2-9581-763b03d73542\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc"
Sep 08 13:33:35.174779 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:35.174689    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc"
Sep 08 13:33:35.212220 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:35.212162    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 08 13:33:35.456585 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:35.456558    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc"]
Sep 08 13:33:35.458967 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:33:35.458931    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc4a75e_814d_4ea2_9581_763b03d73542.slice/crio-a3a92bb55a0d0c91b4a1a90b0f2a2818ffa50134824c29a3203201479e15d72e WatchSource:0}: Error finding container a3a92bb55a0d0c91b4a1a90b0f2a2818ffa50134824c29a3203201479e15d72e: Status 404 returned error can't find the container with id a3a92bb55a0d0c91b4a1a90b0f2a2818ffa50134824c29a3203201479e15d72e
Sep 08 13:33:35.462994 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:35.461560    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:33:35.504938 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:35.504898    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 08 13:33:35.506619 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:33:35.506582    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b1d2f70_7b93_4397_9d5f_c533a6c5b075.slice/crio-c2c9f7788647ec6aa8ef171ee3ed535047672fa7c2a0312d66bbe13dea1c4956 WatchSource:0}: Error finding container c2c9f7788647ec6aa8ef171ee3ed535047672fa7c2a0312d66bbe13dea1c4956: Status 404 returned error can't find the container with id c2c9f7788647ec6aa8ef171ee3ed535047672fa7c2a0312d66bbe13dea1c4956
Sep 08 13:33:35.886663 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:35.886624    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc" event={"ID":"cfc4a75e-814d-4ea2-9581-763b03d73542","Type":"ContainerStarted","Data":"a3a92bb55a0d0c91b4a1a90b0f2a2818ffa50134824c29a3203201479e15d72e"}
Sep 08 13:33:35.888412 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:35.888371    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"4b1d2f70-7b93-4397-9d5f-c533a6c5b075","Type":"ContainerStarted","Data":"c36f3240b88148b7270fcf55d26becbbb118de270c1b76abb003f67b75ad3b7e"}
Sep 08 13:33:35.888553 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:35.888430    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"4b1d2f70-7b93-4397-9d5f-c533a6c5b075","Type":"ContainerStarted","Data":"c2c9f7788647ec6aa8ef171ee3ed535047672fa7c2a0312d66bbe13dea1c4956"}
Sep 08 13:33:37.903627 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:37.903566    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc" event={"ID":"cfc4a75e-814d-4ea2-9581-763b03d73542","Type":"ContainerStarted","Data":"fd84764839d53be0f1063cd99576306b06a07763b841d7fac321c574c331b470"}
Sep 08 13:33:37.904074 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:37.903688    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc"
Sep 08 13:33:37.924709 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:37.924651    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc" podStartSLOduration=1.5730966199999998 podStartE2EDuration="3.924633318s" totalImagesPullingTime="2.351536698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:33:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:33:37.923418753 +0000 UTC m=+1081.375179686" watchObservedRunningTime="2026-09-08 13:33:37.924633318 +0000 UTC m=+1081.376394249"
Sep 08 13:33:38.910356 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:38.910320    2558 generic.go:356] "Generic (PLEG): container finished" podID="4b1d2f70-7b93-4397-9d5f-c533a6c5b075" containerID="c36f3240b88148b7270fcf55d26becbbb118de270c1b76abb003f67b75ad3b7e" exitCode=0
Sep 08 13:33:38.910836 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:38.910396    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"4b1d2f70-7b93-4397-9d5f-c533a6c5b075","Type":"ContainerDied","Data":"c36f3240b88148b7270fcf55d26becbbb118de270c1b76abb003f67b75ad3b7e"}
Sep 08 13:33:39.917993 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:39.917960    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"4b1d2f70-7b93-4397-9d5f-c533a6c5b075","Type":"ContainerStarted","Data":"fb22f224feb8151ed51c2a727747979348272970e7b7a9821f86c8f812618af1"}
Sep 08 13:33:39.939865 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:39.939809    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=3.3654615310000002 podStartE2EDuration="5.939789847s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.574328316s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:33:39.937460342 +0000 UTC m=+1083.389221309" watchObservedRunningTime="2026-09-08 13:33:39.939789847 +0000 UTC m=+1083.391550777"
Sep 08 13:33:40.213144 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:40.213062    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 08 13:33:45.212770 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:45.212716    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 08 13:33:45.214365 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:45.214330    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="4b1d2f70-7b93-4397-9d5f-c533a6c5b075" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
Sep 08 13:33:55.213380 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:33:55.213335    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="4b1d2f70-7b93-4397-9d5f-c533a6c5b075" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
Sep 08 13:34:05.212734 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:34:05.212688    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="4b1d2f70-7b93-4397-9d5f-c533a6c5b075" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
Sep 08 13:34:08.912093 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:34:08.912058    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc"
Sep 08 13:34:15.212902 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:34:15.212854    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="4b1d2f70-7b93-4397-9d5f-c533a6c5b075" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
Sep 08 13:34:25.213136 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:34:25.213093    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="4b1d2f70-7b93-4397-9d5f-c533a6c5b075" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
Sep 08 13:34:35.212845 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:34:35.212803    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="4b1d2f70-7b93-4397-9d5f-c533a6c5b075" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
Sep 08 13:34:45.213056 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:34:45.213009    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="4b1d2f70-7b93-4397-9d5f-c533a6c5b075" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
Sep 08 13:34:55.213233 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:34:55.213187    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="4b1d2f70-7b93-4397-9d5f-c533a6c5b075" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
Sep 08 13:35:05.212633 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:05.212527    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="4b1d2f70-7b93-4397-9d5f-c533a6c5b075" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
Sep 08 13:35:15.223146 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:15.223109    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 08 13:35:15.230946 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:15.230919    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 08 13:35:20.886656 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:20.886621    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc"]
Sep 08 13:35:20.898122 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:20.898091    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 08 13:35:28.912632 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:28.912574    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc" podUID="cfc4a75e-814d-4ea2-9581-763b03d73542" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:35:35.908518 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:35.908452    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc" podUID="cfc4a75e-814d-4ea2-9581-763b03d73542" containerName="main" containerID="cri-o://fd84764839d53be0f1063cd99576306b06a07763b841d7fac321c574c331b470" gracePeriod=45
Sep 08 13:35:35.924754 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:35.924693    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="4b1d2f70-7b93-4397-9d5f-c533a6c5b075" containerName="main" containerID="cri-o://fb22f224feb8151ed51c2a727747979348272970e7b7a9821f86c8f812618af1" gracePeriod=45
Sep 08 13:35:36.224534 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:36.224436    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="4b1d2f70-7b93-4397-9d5f-c533a6c5b075" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": dial tcp 10.132.0.38:8000: connect: connection refused"
Sep 08 13:35:36.682110 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:36.682082    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 08 13:35:36.791418 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:36.791380    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-model-cache\") pod \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\" (UID: \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\") "
Sep 08 13:35:36.791644 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:36.791465    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8kdz\" (UniqueName: \"kubernetes.io/projected/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-kube-api-access-r8kdz\") pod \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\" (UID: \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\") "
Sep 08 13:35:36.791644 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:36.791499    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-kserve-provision-location\") pod \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\" (UID: \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\") "
Sep 08 13:35:36.791644 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:36.791516    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-dshm\") pod \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\" (UID: \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\") "
Sep 08 13:35:36.791644 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:36.791531    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-tmp-dir\") pod \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\" (UID: \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\") "
Sep 08 13:35:36.791887 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:36.791664    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-cabundle-cert\") pod \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\" (UID: \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\") "
Sep 08 13:35:36.791887 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:36.791748    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-home\") pod \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\" (UID: \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\") "
Sep 08 13:35:36.791887 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:36.791788    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-tls-certs\") pod \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\" (UID: \"4b1d2f70-7b93-4397-9d5f-c533a6c5b075\") "
Sep 08 13:35:36.792040 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:36.791663    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-model-cache" (OuterVolumeSpecName: "model-cache") pod "4b1d2f70-7b93-4397-9d5f-c533a6c5b075" (UID: "4b1d2f70-7b93-4397-9d5f-c533a6c5b075"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:35:36.792040 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:36.791986    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4b1d2f70-7b93-4397-9d5f-c533a6c5b075" (UID: "4b1d2f70-7b93-4397-9d5f-c533a6c5b075"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:35:36.792313 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:36.792229    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-cabundle-cert\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:35:36.792313 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:36.792252    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:35:36.792584 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:36.792554    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-home" (OuterVolumeSpecName: "home") pod "4b1d2f70-7b93-4397-9d5f-c533a6c5b075" (UID: "4b1d2f70-7b93-4397-9d5f-c533a6c5b075"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:35:36.794013 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:36.793978    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-dshm" (OuterVolumeSpecName: "dshm") pod "4b1d2f70-7b93-4397-9d5f-c533a6c5b075" (UID: "4b1d2f70-7b93-4397-9d5f-c533a6c5b075"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:35:36.794321 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:36.794298    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-kube-api-access-r8kdz" (OuterVolumeSpecName: "kube-api-access-r8kdz") pod "4b1d2f70-7b93-4397-9d5f-c533a6c5b075" (UID: "4b1d2f70-7b93-4397-9d5f-c533a6c5b075"). InnerVolumeSpecName "kube-api-access-r8kdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:35:36.794402 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:36.794303    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4b1d2f70-7b93-4397-9d5f-c533a6c5b075" (UID: "4b1d2f70-7b93-4397-9d5f-c533a6c5b075"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:35:36.810550 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:36.810512    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4b1d2f70-7b93-4397-9d5f-c533a6c5b075" (UID: "4b1d2f70-7b93-4397-9d5f-c533a6c5b075"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:35:36.815443 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:36.815413    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4b1d2f70-7b93-4397-9d5f-c533a6c5b075" (UID: "4b1d2f70-7b93-4397-9d5f-c533a6c5b075"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:35:36.893255 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:36.893215    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:35:36.893255 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:36.893247    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:35:36.893255 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:36.893258    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8kdz\" (UniqueName: \"kubernetes.io/projected/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-kube-api-access-r8kdz\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:35:36.893488 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:36.893279    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-kserve-provision-location\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:35:36.893488 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:36.893289    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:35:36.893488 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:36.893297    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b1d2f70-7b93-4397-9d5f-c533a6c5b075-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:35:37.540193 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:37.540156    2558 generic.go:356] "Generic (PLEG): container finished" podID="4b1d2f70-7b93-4397-9d5f-c533a6c5b075" containerID="fb22f224feb8151ed51c2a727747979348272970e7b7a9821f86c8f812618af1" exitCode=0
Sep 08 13:35:37.540657 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:37.540204    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 08 13:35:37.540657 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:37.540238    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"4b1d2f70-7b93-4397-9d5f-c533a6c5b075","Type":"ContainerDied","Data":"fb22f224feb8151ed51c2a727747979348272970e7b7a9821f86c8f812618af1"}
Sep 08 13:35:37.540657 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:37.540294    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"4b1d2f70-7b93-4397-9d5f-c533a6c5b075","Type":"ContainerDied","Data":"c2c9f7788647ec6aa8ef171ee3ed535047672fa7c2a0312d66bbe13dea1c4956"}
Sep 08 13:35:37.540657 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:37.540319    2558 scope.go:117] "RemoveContainer" containerID="fb22f224feb8151ed51c2a727747979348272970e7b7a9821f86c8f812618af1"
Sep 08 13:35:37.549061 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:37.549038    2558 scope.go:117] "RemoveContainer" containerID="c36f3240b88148b7270fcf55d26becbbb118de270c1b76abb003f67b75ad3b7e"
Sep 08 13:35:37.556905 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:37.556879    2558 scope.go:117] "RemoveContainer" containerID="fb22f224feb8151ed51c2a727747979348272970e7b7a9821f86c8f812618af1"
Sep 08 13:35:37.557190 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:35:37.557166    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb22f224feb8151ed51c2a727747979348272970e7b7a9821f86c8f812618af1\": container with ID starting with fb22f224feb8151ed51c2a727747979348272970e7b7a9821f86c8f812618af1 not found: ID does not exist" containerID="fb22f224feb8151ed51c2a727747979348272970e7b7a9821f86c8f812618af1"
Sep 08 13:35:37.557261 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:37.557202    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb22f224feb8151ed51c2a727747979348272970e7b7a9821f86c8f812618af1"} err="failed to get container status \"fb22f224feb8151ed51c2a727747979348272970e7b7a9821f86c8f812618af1\": rpc error: code = NotFound desc = could not find container \"fb22f224feb8151ed51c2a727747979348272970e7b7a9821f86c8f812618af1\": container with ID starting with fb22f224feb8151ed51c2a727747979348272970e7b7a9821f86c8f812618af1 not found: ID does not exist"
Sep 08 13:35:37.557261 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:37.557226    2558 scope.go:117] "RemoveContainer" containerID="c36f3240b88148b7270fcf55d26becbbb118de270c1b76abb003f67b75ad3b7e"
Sep 08 13:35:37.557507 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:35:37.557489    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c36f3240b88148b7270fcf55d26becbbb118de270c1b76abb003f67b75ad3b7e\": container with ID starting with c36f3240b88148b7270fcf55d26becbbb118de270c1b76abb003f67b75ad3b7e not found: ID does not exist" containerID="c36f3240b88148b7270fcf55d26becbbb118de270c1b76abb003f67b75ad3b7e"
Sep 08 13:35:37.557557 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:37.557514    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c36f3240b88148b7270fcf55d26becbbb118de270c1b76abb003f67b75ad3b7e"} err="failed to get container status \"c36f3240b88148b7270fcf55d26becbbb118de270c1b76abb003f67b75ad3b7e\": rpc error: code = NotFound desc = could not find container \"c36f3240b88148b7270fcf55d26becbbb118de270c1b76abb003f67b75ad3b7e\": container with ID starting with c36f3240b88148b7270fcf55d26becbbb118de270c1b76abb003f67b75ad3b7e not found: ID does not exist"
Sep 08 13:35:37.558311 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:37.558293    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 08 13:35:37.566558 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:37.566534    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 08 13:35:38.912374 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:38.912339    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc" podUID="cfc4a75e-814d-4ea2-9581-763b03d73542" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:35:39.053180 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:39.053146    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b1d2f70-7b93-4397-9d5f-c533a6c5b075" path="/var/lib/kubelet/pods/4b1d2f70-7b93-4397-9d5f-c533a6c5b075/volumes"
Sep 08 13:35:48.912011 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:48.911967    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc" podUID="cfc4a75e-814d-4ea2-9581-763b03d73542" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:35:48.912507 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:48.912064    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc"
Sep 08 13:35:58.912510 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:35:58.912471    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc" podUID="cfc4a75e-814d-4ea2-9581-763b03d73542" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 08 13:36:06.159120 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:06.159087    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc"
Sep 08 13:36:06.232015 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:06.231918    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dc2j9\" (UniqueName: \"kubernetes.io/projected/cfc4a75e-814d-4ea2-9581-763b03d73542-kube-api-access-dc2j9\") pod \"cfc4a75e-814d-4ea2-9581-763b03d73542\" (UID: \"cfc4a75e-814d-4ea2-9581-763b03d73542\") "
Sep 08 13:36:06.232015 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:06.231985    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc4a75e-814d-4ea2-9581-763b03d73542-tls-certs\") pod \"cfc4a75e-814d-4ea2-9581-763b03d73542\" (UID: \"cfc4a75e-814d-4ea2-9581-763b03d73542\") "
Sep 08 13:36:06.234115 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:06.234084    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfc4a75e-814d-4ea2-9581-763b03d73542-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cfc4a75e-814d-4ea2-9581-763b03d73542" (UID: "cfc4a75e-814d-4ea2-9581-763b03d73542"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:36:06.234227 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:06.234160    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc4a75e-814d-4ea2-9581-763b03d73542-kube-api-access-dc2j9" (OuterVolumeSpecName: "kube-api-access-dc2j9") pod "cfc4a75e-814d-4ea2-9581-763b03d73542" (UID: "cfc4a75e-814d-4ea2-9581-763b03d73542"). InnerVolumeSpecName "kube-api-access-dc2j9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:36:06.332959 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:06.332919    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc4a75e-814d-4ea2-9581-763b03d73542-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:36:06.332959 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:06.332950    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc2j9\" (UniqueName: \"kubernetes.io/projected/cfc4a75e-814d-4ea2-9581-763b03d73542-kube-api-access-dc2j9\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:36:06.679656 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:06.679621    2558 generic.go:356] "Generic (PLEG): container finished" podID="cfc4a75e-814d-4ea2-9581-763b03d73542" containerID="fd84764839d53be0f1063cd99576306b06a07763b841d7fac321c574c331b470" exitCode=0
Sep 08 13:36:06.679853 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:06.679662    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc"
Sep 08 13:36:06.679853 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:06.679707    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc" event={"ID":"cfc4a75e-814d-4ea2-9581-763b03d73542","Type":"ContainerDied","Data":"fd84764839d53be0f1063cd99576306b06a07763b841d7fac321c574c331b470"}
Sep 08 13:36:06.679853 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:06.679760    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc" event={"ID":"cfc4a75e-814d-4ea2-9581-763b03d73542","Type":"ContainerDied","Data":"a3a92bb55a0d0c91b4a1a90b0f2a2818ffa50134824c29a3203201479e15d72e"}
Sep 08 13:36:06.679853 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:06.679795    2558 scope.go:117] "RemoveContainer" containerID="fd84764839d53be0f1063cd99576306b06a07763b841d7fac321c574c331b470"
Sep 08 13:36:06.688754 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:06.688732    2558 scope.go:117] "RemoveContainer" containerID="fd84764839d53be0f1063cd99576306b06a07763b841d7fac321c574c331b470"
Sep 08 13:36:06.689041 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:36:06.689020    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd84764839d53be0f1063cd99576306b06a07763b841d7fac321c574c331b470\": container with ID starting with fd84764839d53be0f1063cd99576306b06a07763b841d7fac321c574c331b470 not found: ID does not exist" containerID="fd84764839d53be0f1063cd99576306b06a07763b841d7fac321c574c331b470"
Sep 08 13:36:06.689097 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:06.689050    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd84764839d53be0f1063cd99576306b06a07763b841d7fac321c574c331b470"} err="failed to get container status \"fd84764839d53be0f1063cd99576306b06a07763b841d7fac321c574c331b470\": rpc error: code = NotFound desc = could not find container \"fd84764839d53be0f1063cd99576306b06a07763b841d7fac321c574c331b470\": container with ID starting with fd84764839d53be0f1063cd99576306b06a07763b841d7fac321c574c331b470 not found: ID does not exist"
Sep 08 13:36:06.700225 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:06.700200    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc"]
Sep 08 13:36:06.703641 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:06.703617    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schenm4cc"]
Sep 08 13:36:07.054014 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:07.053935    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfc4a75e-814d-4ea2-9581-763b03d73542" path="/var/lib/kubelet/pods/cfc4a75e-814d-4ea2-9581-763b03d73542/volumes"
Sep 08 13:36:15.343648 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.343610    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd"]
Sep 08 13:36:15.344074 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.343987    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfc4a75e-814d-4ea2-9581-763b03d73542" containerName="main"
Sep 08 13:36:15.344074 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.343999    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfc4a75e-814d-4ea2-9581-763b03d73542" containerName="main"
Sep 08 13:36:15.344074 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.344012    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b1d2f70-7b93-4397-9d5f-c533a6c5b075" containerName="main"
Sep 08 13:36:15.344074 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.344017    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b1d2f70-7b93-4397-9d5f-c533a6c5b075" containerName="main"
Sep 08 13:36:15.344074 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.344031    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b1d2f70-7b93-4397-9d5f-c533a6c5b075" containerName="storage-initializer"
Sep 08 13:36:15.344074 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.344038    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b1d2f70-7b93-4397-9d5f-c533a6c5b075" containerName="storage-initializer"
Sep 08 13:36:15.344259 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.344107    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b1d2f70-7b93-4397-9d5f-c533a6c5b075" containerName="main"
Sep 08 13:36:15.344259 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.344115    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfc4a75e-814d-4ea2-9581-763b03d73542" containerName="main"
Sep 08 13:36:15.346985 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.346953    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd"
Sep 08 13:36:15.351529 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.351503    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 08 13:36:15.351710 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.351579    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 08 13:36:15.351710 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.351680    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 08 13:36:15.351852 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.351736    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-mx8zh\""
Sep 08 13:36:15.370314 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.370283    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd"]
Sep 08 13:36:15.409149 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.409111    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c81e9a09-2255-4d0e-b254-de02201ad04a-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-czbrd\" (UID: \"c81e9a09-2255-4d0e-b254-de02201ad04a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd"
Sep 08 13:36:15.409149 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.409156    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c81e9a09-2255-4d0e-b254-de02201ad04a-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-czbrd\" (UID: \"c81e9a09-2255-4d0e-b254-de02201ad04a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd"
Sep 08 13:36:15.409409 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.409271    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c81e9a09-2255-4d0e-b254-de02201ad04a-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-czbrd\" (UID: \"c81e9a09-2255-4d0e-b254-de02201ad04a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd"
Sep 08 13:36:15.409409 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.409331    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzj6g\" (UniqueName: \"kubernetes.io/projected/c81e9a09-2255-4d0e-b254-de02201ad04a-kube-api-access-gzj6g\") pod \"scheduler-inline-config-test-kserve-8655bc757-czbrd\" (UID: \"c81e9a09-2255-4d0e-b254-de02201ad04a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd"
Sep 08 13:36:15.409409 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.409369    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c81e9a09-2255-4d0e-b254-de02201ad04a-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-czbrd\" (UID: \"c81e9a09-2255-4d0e-b254-de02201ad04a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd"
Sep 08 13:36:15.409531 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.409446    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c81e9a09-2255-4d0e-b254-de02201ad04a-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-czbrd\" (UID: \"c81e9a09-2255-4d0e-b254-de02201ad04a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd"
Sep 08 13:36:15.509908 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.509869    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c81e9a09-2255-4d0e-b254-de02201ad04a-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-czbrd\" (UID: \"c81e9a09-2255-4d0e-b254-de02201ad04a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd"
Sep 08 13:36:15.509908 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.509911    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c81e9a09-2255-4d0e-b254-de02201ad04a-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-czbrd\" (UID: \"c81e9a09-2255-4d0e-b254-de02201ad04a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd"
Sep 08 13:36:15.510175 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.509929    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c81e9a09-2255-4d0e-b254-de02201ad04a-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-czbrd\" (UID: \"c81e9a09-2255-4d0e-b254-de02201ad04a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd"
Sep 08 13:36:15.510175 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.509979    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c81e9a09-2255-4d0e-b254-de02201ad04a-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-czbrd\" (UID: \"c81e9a09-2255-4d0e-b254-de02201ad04a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd"
Sep 08 13:36:15.510175 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.510014    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzj6g\" (UniqueName: \"kubernetes.io/projected/c81e9a09-2255-4d0e-b254-de02201ad04a-kube-api-access-gzj6g\") pod \"scheduler-inline-config-test-kserve-8655bc757-czbrd\" (UID: \"c81e9a09-2255-4d0e-b254-de02201ad04a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd"
Sep 08 13:36:15.510175 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.510046    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c81e9a09-2255-4d0e-b254-de02201ad04a-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-czbrd\" (UID: \"c81e9a09-2255-4d0e-b254-de02201ad04a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd"
Sep 08 13:36:15.510419 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.510345    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c81e9a09-2255-4d0e-b254-de02201ad04a-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-czbrd\" (UID: \"c81e9a09-2255-4d0e-b254-de02201ad04a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd"
Sep 08 13:36:15.510419 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.510394    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c81e9a09-2255-4d0e-b254-de02201ad04a-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-czbrd\" (UID: \"c81e9a09-2255-4d0e-b254-de02201ad04a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd"
Sep 08 13:36:15.510520 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.510434    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c81e9a09-2255-4d0e-b254-de02201ad04a-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-czbrd\" (UID: \"c81e9a09-2255-4d0e-b254-de02201ad04a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd"
Sep 08 13:36:15.512305 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.512278    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c81e9a09-2255-4d0e-b254-de02201ad04a-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-czbrd\" (UID: \"c81e9a09-2255-4d0e-b254-de02201ad04a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd"
Sep 08 13:36:15.512545 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.512528    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c81e9a09-2255-4d0e-b254-de02201ad04a-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-czbrd\" (UID: \"c81e9a09-2255-4d0e-b254-de02201ad04a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd"
Sep 08 13:36:15.519863 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.519837    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzj6g\" (UniqueName: \"kubernetes.io/projected/c81e9a09-2255-4d0e-b254-de02201ad04a-kube-api-access-gzj6g\") pod \"scheduler-inline-config-test-kserve-8655bc757-czbrd\" (UID: \"c81e9a09-2255-4d0e-b254-de02201ad04a\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd"
Sep 08 13:36:15.657647 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.657612    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd"
Sep 08 13:36:15.934650 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:15.934625    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd"]
Sep 08 13:36:15.936756 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:36:15.936725    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc81e9a09_2255_4d0e_b254_de02201ad04a.slice/crio-b2beab70713e58e84bcbf2d64bbd03943e12adf6ac07713033df4a5907c2fc1d WatchSource:0}: Error finding container b2beab70713e58e84bcbf2d64bbd03943e12adf6ac07713033df4a5907c2fc1d: Status 404 returned error can't find the container with id b2beab70713e58e84bcbf2d64bbd03943e12adf6ac07713033df4a5907c2fc1d
Sep 08 13:36:16.736313 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:16.736234    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd" event={"ID":"c81e9a09-2255-4d0e-b254-de02201ad04a","Type":"ContainerStarted","Data":"b2beab70713e58e84bcbf2d64bbd03943e12adf6ac07713033df4a5907c2fc1d"}
Sep 08 13:36:17.742546 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:17.742510    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd" event={"ID":"c81e9a09-2255-4d0e-b254-de02201ad04a","Type":"ContainerStarted","Data":"91a9848cb2b37ab958b40cf4d1e3855be1129b2897e477f8569afab2b83b01fd"}
Sep 08 13:36:17.765375 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:17.765318    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd" podStartSLOduration=1.144381079 podStartE2EDuration="2.765302207s" totalImagesPullingTime="1.620921128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:36:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:36:17.764173454 +0000 UTC m=+1241.215934387" watchObservedRunningTime="2026-09-08 13:36:17.765302207 +0000 UTC m=+1241.217063137"
Sep 08 13:36:18.657892 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:18.657854    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd"
Sep 08 13:36:25.658775 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:25.658732    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd"
Sep 08 13:36:25.671354 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:25.671327    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd"
Sep 08 13:36:25.791302 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:25.791267    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd"
Sep 08 13:36:49.895426 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:49.895394    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd"]
Sep 08 13:36:49.906696 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:36:49.906643    2558 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 08 13:36:49.906696 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:36:49.906696 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:36:49.906696 ip-10-0-128-147 kubenswrapper[2558]:  > containerID="91a9848cb2b37ab958b40cf4d1e3855be1129b2897e477f8569afab2b83b01fd" cmd=["/bin/sleep","15"]
Sep 08 13:36:49.907113 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:49.907079    2558 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd" message="" err=<
Sep 08 13:36:49.907113 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:36:49.907113 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:36:49.907113 ip-10-0-128-147 kubenswrapper[2558]:  >
Sep 08 13:36:49.907394 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:36:49.907363    2558 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 08 13:36:49.907394 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:36:49.907394 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:36:49.907394 ip-10-0-128-147 kubenswrapper[2558]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd" podUID="c81e9a09-2255-4d0e-b254-de02201ad04a" containerName="main" containerID="cri-o://91a9848cb2b37ab958b40cf4d1e3855be1129b2897e477f8569afab2b83b01fd"
Sep 08 13:36:49.907709 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:49.907681    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd" podUID="c81e9a09-2255-4d0e-b254-de02201ad04a" containerName="main" containerID="cri-o://91a9848cb2b37ab958b40cf4d1e3855be1129b2897e477f8569afab2b83b01fd" gracePeriod=60
Sep 08 13:36:50.659739 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:50.659714    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd"
Sep 08 13:36:50.808817 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:50.808734    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzj6g\" (UniqueName: \"kubernetes.io/projected/c81e9a09-2255-4d0e-b254-de02201ad04a-kube-api-access-gzj6g\") pod \"c81e9a09-2255-4d0e-b254-de02201ad04a\" (UID: \"c81e9a09-2255-4d0e-b254-de02201ad04a\") "
Sep 08 13:36:50.808817 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:50.808781    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c81e9a09-2255-4d0e-b254-de02201ad04a-tmp-dir\") pod \"c81e9a09-2255-4d0e-b254-de02201ad04a\" (UID: \"c81e9a09-2255-4d0e-b254-de02201ad04a\") "
Sep 08 13:36:50.808817 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:50.808802    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c81e9a09-2255-4d0e-b254-de02201ad04a-tls-certs\") pod \"c81e9a09-2255-4d0e-b254-de02201ad04a\" (UID: \"c81e9a09-2255-4d0e-b254-de02201ad04a\") "
Sep 08 13:36:50.809072 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:50.808826    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c81e9a09-2255-4d0e-b254-de02201ad04a-home\") pod \"c81e9a09-2255-4d0e-b254-de02201ad04a\" (UID: \"c81e9a09-2255-4d0e-b254-de02201ad04a\") "
Sep 08 13:36:50.809072 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:50.808859    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c81e9a09-2255-4d0e-b254-de02201ad04a-dshm\") pod \"c81e9a09-2255-4d0e-b254-de02201ad04a\" (UID: \"c81e9a09-2255-4d0e-b254-de02201ad04a\") "
Sep 08 13:36:50.809072 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:50.808882    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c81e9a09-2255-4d0e-b254-de02201ad04a-model-cache\") pod \"c81e9a09-2255-4d0e-b254-de02201ad04a\" (UID: \"c81e9a09-2255-4d0e-b254-de02201ad04a\") "
Sep 08 13:36:50.809231 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:50.809057    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81e9a09-2255-4d0e-b254-de02201ad04a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c81e9a09-2255-4d0e-b254-de02201ad04a" (UID: "c81e9a09-2255-4d0e-b254-de02201ad04a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:36:50.809231 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:50.809123    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81e9a09-2255-4d0e-b254-de02201ad04a-home" (OuterVolumeSpecName: "home") pod "c81e9a09-2255-4d0e-b254-de02201ad04a" (UID: "c81e9a09-2255-4d0e-b254-de02201ad04a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:36:50.809318 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:50.809223    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81e9a09-2255-4d0e-b254-de02201ad04a-model-cache" (OuterVolumeSpecName: "model-cache") pod "c81e9a09-2255-4d0e-b254-de02201ad04a" (UID: "c81e9a09-2255-4d0e-b254-de02201ad04a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:36:50.811044 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:50.811017    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81e9a09-2255-4d0e-b254-de02201ad04a-dshm" (OuterVolumeSpecName: "dshm") pod "c81e9a09-2255-4d0e-b254-de02201ad04a" (UID: "c81e9a09-2255-4d0e-b254-de02201ad04a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:36:50.811044 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:50.811033    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81e9a09-2255-4d0e-b254-de02201ad04a-kube-api-access-gzj6g" (OuterVolumeSpecName: "kube-api-access-gzj6g") pod "c81e9a09-2255-4d0e-b254-de02201ad04a" (UID: "c81e9a09-2255-4d0e-b254-de02201ad04a"). InnerVolumeSpecName "kube-api-access-gzj6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:36:50.811183 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:50.811103    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c81e9a09-2255-4d0e-b254-de02201ad04a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c81e9a09-2255-4d0e-b254-de02201ad04a" (UID: "c81e9a09-2255-4d0e-b254-de02201ad04a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:36:50.899028 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:50.898994    2558 generic.go:356] "Generic (PLEG): container finished" podID="c81e9a09-2255-4d0e-b254-de02201ad04a" containerID="91a9848cb2b37ab958b40cf4d1e3855be1129b2897e477f8569afab2b83b01fd" exitCode=0
Sep 08 13:36:50.899028 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:50.899035    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd" event={"ID":"c81e9a09-2255-4d0e-b254-de02201ad04a","Type":"ContainerDied","Data":"91a9848cb2b37ab958b40cf4d1e3855be1129b2897e477f8569afab2b83b01fd"}
Sep 08 13:36:50.899557 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:50.899047    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd"
Sep 08 13:36:50.899557 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:50.899078    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd" event={"ID":"c81e9a09-2255-4d0e-b254-de02201ad04a","Type":"ContainerDied","Data":"b2beab70713e58e84bcbf2d64bbd03943e12adf6ac07713033df4a5907c2fc1d"}
Sep 08 13:36:50.899557 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:50.899100    2558 scope.go:117] "RemoveContainer" containerID="91a9848cb2b37ab958b40cf4d1e3855be1129b2897e477f8569afab2b83b01fd"
Sep 08 13:36:50.908402 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:50.908384    2558 scope.go:117] "RemoveContainer" containerID="91a9848cb2b37ab958b40cf4d1e3855be1129b2897e477f8569afab2b83b01fd"
Sep 08 13:36:50.908660 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:36:50.908641    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91a9848cb2b37ab958b40cf4d1e3855be1129b2897e477f8569afab2b83b01fd\": container with ID starting with 91a9848cb2b37ab958b40cf4d1e3855be1129b2897e477f8569afab2b83b01fd not found: ID does not exist" containerID="91a9848cb2b37ab958b40cf4d1e3855be1129b2897e477f8569afab2b83b01fd"
Sep 08 13:36:50.908720 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:50.908668    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91a9848cb2b37ab958b40cf4d1e3855be1129b2897e477f8569afab2b83b01fd"} err="failed to get container status \"91a9848cb2b37ab958b40cf4d1e3855be1129b2897e477f8569afab2b83b01fd\": rpc error: code = NotFound desc = could not find container \"91a9848cb2b37ab958b40cf4d1e3855be1129b2897e477f8569afab2b83b01fd\": container with ID starting with 91a9848cb2b37ab958b40cf4d1e3855be1129b2897e477f8569afab2b83b01fd not found: ID does not exist"
Sep 08 13:36:50.910066 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:50.910048    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c81e9a09-2255-4d0e-b254-de02201ad04a-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:36:50.910112 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:50.910069    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzj6g\" (UniqueName: \"kubernetes.io/projected/c81e9a09-2255-4d0e-b254-de02201ad04a-kube-api-access-gzj6g\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:36:50.910112 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:50.910079    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c81e9a09-2255-4d0e-b254-de02201ad04a-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:36:50.910112 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:50.910087    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c81e9a09-2255-4d0e-b254-de02201ad04a-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:36:50.910112 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:50.910094    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c81e9a09-2255-4d0e-b254-de02201ad04a-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:36:50.910112 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:50.910102    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c81e9a09-2255-4d0e-b254-de02201ad04a-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:36:50.921654 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:50.921624    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd"]
Sep 08 13:36:50.923803 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:50.923779    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-czbrd"]
Sep 08 13:36:51.054005 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:36:51.053969    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c81e9a09-2255-4d0e-b254-de02201ad04a" path="/var/lib/kubelet/pods/c81e9a09-2255-4d0e-b254-de02201ad04a/volumes"
Sep 08 13:38:01.414718 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:01.414564    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx"]
Sep 08 13:38:01.415148 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:01.414925    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c81e9a09-2255-4d0e-b254-de02201ad04a" containerName="main"
Sep 08 13:38:01.415148 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:01.414940    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c81e9a09-2255-4d0e-b254-de02201ad04a" containerName="main"
Sep 08 13:38:01.415148 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:01.415012    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c81e9a09-2255-4d0e-b254-de02201ad04a" containerName="main"
Sep 08 13:38:01.419085 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:01.419065    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx"
Sep 08 13:38:01.421582 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:01.421561    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 08 13:38:01.422490 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:01.422474    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 08 13:38:01.422548 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:01.422517    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 08 13:38:01.422548 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:01.422543    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-d6l9g\""
Sep 08 13:38:01.428760 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:01.428736    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx"]
Sep 08 13:38:01.507079 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:01.507044    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bbbb790-b1f2-46de-91c1-625c5be74407-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx\" (UID: \"4bbbb790-b1f2-46de-91c1-625c5be74407\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx"
Sep 08 13:38:01.507317 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:01.507094    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bbbb790-b1f2-46de-91c1-625c5be74407-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx\" (UID: \"4bbbb790-b1f2-46de-91c1-625c5be74407\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx"
Sep 08 13:38:01.507317 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:01.507123    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgh4s\" (UniqueName: \"kubernetes.io/projected/4bbbb790-b1f2-46de-91c1-625c5be74407-kube-api-access-lgh4s\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx\" (UID: \"4bbbb790-b1f2-46de-91c1-625c5be74407\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx"
Sep 08 13:38:01.507317 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:01.507148    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bbbb790-b1f2-46de-91c1-625c5be74407-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx\" (UID: \"4bbbb790-b1f2-46de-91c1-625c5be74407\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx"
Sep 08 13:38:01.507317 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:01.507175    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bbbb790-b1f2-46de-91c1-625c5be74407-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx\" (UID: \"4bbbb790-b1f2-46de-91c1-625c5be74407\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx"
Sep 08 13:38:01.507317 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:01.507257    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bbbb790-b1f2-46de-91c1-625c5be74407-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx\" (UID: \"4bbbb790-b1f2-46de-91c1-625c5be74407\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx"
Sep 08 13:38:01.608387 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:01.608355    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bbbb790-b1f2-46de-91c1-625c5be74407-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx\" (UID: \"4bbbb790-b1f2-46de-91c1-625c5be74407\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx"
Sep 08 13:38:01.608579 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:01.608412    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bbbb790-b1f2-46de-91c1-625c5be74407-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx\" (UID: \"4bbbb790-b1f2-46de-91c1-625c5be74407\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx"
Sep 08 13:38:01.608579 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:01.608445    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bbbb790-b1f2-46de-91c1-625c5be74407-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx\" (UID: \"4bbbb790-b1f2-46de-91c1-625c5be74407\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx"
Sep 08 13:38:01.608579 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:01.608462    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgh4s\" (UniqueName: \"kubernetes.io/projected/4bbbb790-b1f2-46de-91c1-625c5be74407-kube-api-access-lgh4s\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx\" (UID: \"4bbbb790-b1f2-46de-91c1-625c5be74407\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx"
Sep 08 13:38:01.608579 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:01.608545    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bbbb790-b1f2-46de-91c1-625c5be74407-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx\" (UID: \"4bbbb790-b1f2-46de-91c1-625c5be74407\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx"
Sep 08 13:38:01.608836 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:01.608581    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bbbb790-b1f2-46de-91c1-625c5be74407-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx\" (UID: \"4bbbb790-b1f2-46de-91c1-625c5be74407\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx"
Sep 08 13:38:01.608900 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:01.608872    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bbbb790-b1f2-46de-91c1-625c5be74407-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx\" (UID: \"4bbbb790-b1f2-46de-91c1-625c5be74407\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx"
Sep 08 13:38:01.608979 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:01.608958    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bbbb790-b1f2-46de-91c1-625c5be74407-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx\" (UID: \"4bbbb790-b1f2-46de-91c1-625c5be74407\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx"
Sep 08 13:38:01.609106 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:01.609087    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bbbb790-b1f2-46de-91c1-625c5be74407-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx\" (UID: \"4bbbb790-b1f2-46de-91c1-625c5be74407\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx"
Sep 08 13:38:01.610793 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:01.610773    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bbbb790-b1f2-46de-91c1-625c5be74407-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx\" (UID: \"4bbbb790-b1f2-46de-91c1-625c5be74407\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx"
Sep 08 13:38:01.610945 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:01.610928    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bbbb790-b1f2-46de-91c1-625c5be74407-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx\" (UID: \"4bbbb790-b1f2-46de-91c1-625c5be74407\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx"
Sep 08 13:38:01.617684 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:01.617652    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgh4s\" (UniqueName: \"kubernetes.io/projected/4bbbb790-b1f2-46de-91c1-625c5be74407-kube-api-access-lgh4s\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx\" (UID: \"4bbbb790-b1f2-46de-91c1-625c5be74407\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx"
Sep 08 13:38:01.730025 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:01.729936    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx"
Sep 08 13:38:02.008794 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:02.008767    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx"]
Sep 08 13:38:02.011700 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:38:02.011669    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bbbb790_b1f2_46de_91c1_625c5be74407.slice/crio-0c77f247ff0877acf6900c600599b1e1f55a87510f0e2692dcc26f90915b0c51 WatchSource:0}: Error finding container 0c77f247ff0877acf6900c600599b1e1f55a87510f0e2692dcc26f90915b0c51: Status 404 returned error can't find the container with id 0c77f247ff0877acf6900c600599b1e1f55a87510f0e2692dcc26f90915b0c51
Sep 08 13:38:02.238096 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:02.238062    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx" event={"ID":"4bbbb790-b1f2-46de-91c1-625c5be74407","Type":"ContainerStarted","Data":"71c2ad0387f9ac78994261b5cbf8fd393bf5a1fca632953623157cc634a308f9"}
Sep 08 13:38:02.238265 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:02.238114    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx" event={"ID":"4bbbb790-b1f2-46de-91c1-625c5be74407","Type":"ContainerStarted","Data":"0c77f247ff0877acf6900c600599b1e1f55a87510f0e2692dcc26f90915b0c51"}
Sep 08 13:38:02.263263 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:02.263154    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx" podStartSLOduration=1.2631325709999999 podStartE2EDuration="1.263132571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:38:02.262862397 +0000 UTC m=+1345.714623324" watchObservedRunningTime="2026-09-08 13:38:02.263132571 +0000 UTC m=+1345.714893503"
Sep 08 13:38:02.730050 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:02.730006    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx"
Sep 08 13:38:11.730730 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:11.730690    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx"
Sep 08 13:38:11.743227 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:11.743193    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx"
Sep 08 13:38:12.295684 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:12.295651    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx"
Sep 08 13:38:35.093195 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:35.093162    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx"]
Sep 08 13:38:35.102866 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:38:35.102824    2558 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 08 13:38:35.102866 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:38:35.102866 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:38:35.102866 ip-10-0-128-147 kubenswrapper[2558]:  > containerID="71c2ad0387f9ac78994261b5cbf8fd393bf5a1fca632953623157cc634a308f9" cmd=["/bin/sleep","15"]
Sep 08 13:38:35.103095 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:35.102893    2558 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx" message="" err=<
Sep 08 13:38:35.103095 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:38:35.103095 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:38:35.103095 ip-10-0-128-147 kubenswrapper[2558]:  >
Sep 08 13:38:35.103095 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:38:35.102912    2558 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 08 13:38:35.103095 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:38:35.103095 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:38:35.103095 ip-10-0-128-147 kubenswrapper[2558]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx" podUID="4bbbb790-b1f2-46de-91c1-625c5be74407" containerName="main" containerID="cri-o://71c2ad0387f9ac78994261b5cbf8fd393bf5a1fca632953623157cc634a308f9"
Sep 08 13:38:35.103095 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:35.102953    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx" podUID="4bbbb790-b1f2-46de-91c1-625c5be74407" containerName="main" containerID="cri-o://71c2ad0387f9ac78994261b5cbf8fd393bf5a1fca632953623157cc634a308f9" gracePeriod=60
Sep 08 13:38:35.285799 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:35.285754    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx" podUID="4bbbb790-b1f2-46de-91c1-625c5be74407" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Sep 08 13:38:35.402039 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:35.402006    2558 generic.go:356] "Generic (PLEG): container finished" podID="4bbbb790-b1f2-46de-91c1-625c5be74407" containerID="71c2ad0387f9ac78994261b5cbf8fd393bf5a1fca632953623157cc634a308f9" exitCode=0
Sep 08 13:38:35.402203 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:35.402088    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx" event={"ID":"4bbbb790-b1f2-46de-91c1-625c5be74407","Type":"ContainerDied","Data":"71c2ad0387f9ac78994261b5cbf8fd393bf5a1fca632953623157cc634a308f9"}
Sep 08 13:38:35.851116 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:35.851091    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx"
Sep 08 13:38:36.018029 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:36.017943    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bbbb790-b1f2-46de-91c1-625c5be74407-home\") pod \"4bbbb790-b1f2-46de-91c1-625c5be74407\" (UID: \"4bbbb790-b1f2-46de-91c1-625c5be74407\") "
Sep 08 13:38:36.018029 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:36.017981    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bbbb790-b1f2-46de-91c1-625c5be74407-dshm\") pod \"4bbbb790-b1f2-46de-91c1-625c5be74407\" (UID: \"4bbbb790-b1f2-46de-91c1-625c5be74407\") "
Sep 08 13:38:36.018029 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:36.018015    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bbbb790-b1f2-46de-91c1-625c5be74407-model-cache\") pod \"4bbbb790-b1f2-46de-91c1-625c5be74407\" (UID: \"4bbbb790-b1f2-46de-91c1-625c5be74407\") "
Sep 08 13:38:36.018314 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:36.018070    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgh4s\" (UniqueName: \"kubernetes.io/projected/4bbbb790-b1f2-46de-91c1-625c5be74407-kube-api-access-lgh4s\") pod \"4bbbb790-b1f2-46de-91c1-625c5be74407\" (UID: \"4bbbb790-b1f2-46de-91c1-625c5be74407\") "
Sep 08 13:38:36.018314 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:36.018097    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bbbb790-b1f2-46de-91c1-625c5be74407-tls-certs\") pod \"4bbbb790-b1f2-46de-91c1-625c5be74407\" (UID: \"4bbbb790-b1f2-46de-91c1-625c5be74407\") "
Sep 08 13:38:36.018314 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:36.018257    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bbbb790-b1f2-46de-91c1-625c5be74407-tmp-dir\") pod \"4bbbb790-b1f2-46de-91c1-625c5be74407\" (UID: \"4bbbb790-b1f2-46de-91c1-625c5be74407\") "
Sep 08 13:38:36.018448 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:36.018262    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bbbb790-b1f2-46de-91c1-625c5be74407-home" (OuterVolumeSpecName: "home") pod "4bbbb790-b1f2-46de-91c1-625c5be74407" (UID: "4bbbb790-b1f2-46de-91c1-625c5be74407"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:38:36.018448 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:36.018339    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bbbb790-b1f2-46de-91c1-625c5be74407-model-cache" (OuterVolumeSpecName: "model-cache") pod "4bbbb790-b1f2-46de-91c1-625c5be74407" (UID: "4bbbb790-b1f2-46de-91c1-625c5be74407"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:38:36.018565 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:36.018486    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bbbb790-b1f2-46de-91c1-625c5be74407-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4bbbb790-b1f2-46de-91c1-625c5be74407" (UID: "4bbbb790-b1f2-46de-91c1-625c5be74407"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:38:36.018565 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:36.018537    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bbbb790-b1f2-46de-91c1-625c5be74407-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:38:36.018565 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:36.018552    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bbbb790-b1f2-46de-91c1-625c5be74407-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:38:36.020384 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:36.020351    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bbbb790-b1f2-46de-91c1-625c5be74407-dshm" (OuterVolumeSpecName: "dshm") pod "4bbbb790-b1f2-46de-91c1-625c5be74407" (UID: "4bbbb790-b1f2-46de-91c1-625c5be74407"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:38:36.020490 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:36.020415    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bbbb790-b1f2-46de-91c1-625c5be74407-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4bbbb790-b1f2-46de-91c1-625c5be74407" (UID: "4bbbb790-b1f2-46de-91c1-625c5be74407"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:38:36.020490 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:36.020448    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bbbb790-b1f2-46de-91c1-625c5be74407-kube-api-access-lgh4s" (OuterVolumeSpecName: "kube-api-access-lgh4s") pod "4bbbb790-b1f2-46de-91c1-625c5be74407" (UID: "4bbbb790-b1f2-46de-91c1-625c5be74407"). InnerVolumeSpecName "kube-api-access-lgh4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:38:36.119185 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:36.119130    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bbbb790-b1f2-46de-91c1-625c5be74407-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:38:36.119185 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:36.119177    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgh4s\" (UniqueName: \"kubernetes.io/projected/4bbbb790-b1f2-46de-91c1-625c5be74407-kube-api-access-lgh4s\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:38:36.119185 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:36.119193    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bbbb790-b1f2-46de-91c1-625c5be74407-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:38:36.119669 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:36.119206    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bbbb790-b1f2-46de-91c1-625c5be74407-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:38:36.407940 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:36.407906    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx" event={"ID":"4bbbb790-b1f2-46de-91c1-625c5be74407","Type":"ContainerDied","Data":"0c77f247ff0877acf6900c600599b1e1f55a87510f0e2692dcc26f90915b0c51"}
Sep 08 13:38:36.407940 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:36.407938    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx"
Sep 08 13:38:36.408198 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:36.407980    2558 scope.go:117] "RemoveContainer" containerID="71c2ad0387f9ac78994261b5cbf8fd393bf5a1fca632953623157cc634a308f9"
Sep 08 13:38:36.429946 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:36.429921    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx"]
Sep 08 13:38:36.434291 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:36.434268    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fz7bfx"]
Sep 08 13:38:37.054422 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:38:37.054390    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bbbb790-b1f2-46de-91c1-625c5be74407" path="/var/lib/kubelet/pods/4bbbb790-b1f2-46de-91c1-625c5be74407/volumes"
Sep 08 13:39:31.740346 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:31.740262    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz"]
Sep 08 13:39:31.740783 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:31.740649    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bbbb790-b1f2-46de-91c1-625c5be74407" containerName="main"
Sep 08 13:39:31.740783 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:31.740661    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bbbb790-b1f2-46de-91c1-625c5be74407" containerName="main"
Sep 08 13:39:31.740783 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:31.740742    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bbbb790-b1f2-46de-91c1-625c5be74407" containerName="main"
Sep 08 13:39:31.743788 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:31.743770    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz"
Sep 08 13:39:31.746405 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:31.746377    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 08 13:39:31.747139 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:31.747121    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 08 13:39:31.747250 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:31.747145    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-9zkkj\""
Sep 08 13:39:31.747250 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:31.747145    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 08 13:39:31.757688 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:31.757665    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz"]
Sep 08 13:39:31.894411 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:31.894371    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d846102d-c04b-4cea-b451-13466d2adbea-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz\" (UID: \"d846102d-c04b-4cea-b451-13466d2adbea\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz"
Sep 08 13:39:31.894769 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:31.894421    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d846102d-c04b-4cea-b451-13466d2adbea-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz\" (UID: \"d846102d-c04b-4cea-b451-13466d2adbea\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz"
Sep 08 13:39:31.894769 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:31.894497    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d846102d-c04b-4cea-b451-13466d2adbea-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz\" (UID: \"d846102d-c04b-4cea-b451-13466d2adbea\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz"
Sep 08 13:39:31.894769 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:31.894540    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnrtb\" (UniqueName: \"kubernetes.io/projected/d846102d-c04b-4cea-b451-13466d2adbea-kube-api-access-tnrtb\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz\" (UID: \"d846102d-c04b-4cea-b451-13466d2adbea\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz"
Sep 08 13:39:31.894769 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:31.894569    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d846102d-c04b-4cea-b451-13466d2adbea-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz\" (UID: \"d846102d-c04b-4cea-b451-13466d2adbea\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz"
Sep 08 13:39:31.894769 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:31.894586    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d846102d-c04b-4cea-b451-13466d2adbea-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz\" (UID: \"d846102d-c04b-4cea-b451-13466d2adbea\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz"
Sep 08 13:39:31.995688 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:31.995577    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d846102d-c04b-4cea-b451-13466d2adbea-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz\" (UID: \"d846102d-c04b-4cea-b451-13466d2adbea\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz"
Sep 08 13:39:31.995971 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:31.995941    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnrtb\" (UniqueName: \"kubernetes.io/projected/d846102d-c04b-4cea-b451-13466d2adbea-kube-api-access-tnrtb\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz\" (UID: \"d846102d-c04b-4cea-b451-13466d2adbea\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz"
Sep 08 13:39:31.996080 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:31.995997    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d846102d-c04b-4cea-b451-13466d2adbea-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz\" (UID: \"d846102d-c04b-4cea-b451-13466d2adbea\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz"
Sep 08 13:39:31.996080 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:31.996024    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d846102d-c04b-4cea-b451-13466d2adbea-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz\" (UID: \"d846102d-c04b-4cea-b451-13466d2adbea\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz"
Sep 08 13:39:31.996193 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:31.996087    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d846102d-c04b-4cea-b451-13466d2adbea-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz\" (UID: \"d846102d-c04b-4cea-b451-13466d2adbea\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz"
Sep 08 13:39:31.996193 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:31.996101    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d846102d-c04b-4cea-b451-13466d2adbea-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz\" (UID: \"d846102d-c04b-4cea-b451-13466d2adbea\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz"
Sep 08 13:39:31.996193 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:31.996170    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d846102d-c04b-4cea-b451-13466d2adbea-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz\" (UID: \"d846102d-c04b-4cea-b451-13466d2adbea\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz"
Sep 08 13:39:31.996445 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:31.996425    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d846102d-c04b-4cea-b451-13466d2adbea-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz\" (UID: \"d846102d-c04b-4cea-b451-13466d2adbea\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz"
Sep 08 13:39:31.996508 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:31.996456    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d846102d-c04b-4cea-b451-13466d2adbea-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz\" (UID: \"d846102d-c04b-4cea-b451-13466d2adbea\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz"
Sep 08 13:39:31.998376 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:31.998353    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d846102d-c04b-4cea-b451-13466d2adbea-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz\" (UID: \"d846102d-c04b-4cea-b451-13466d2adbea\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz"
Sep 08 13:39:31.998666 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:31.998647    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d846102d-c04b-4cea-b451-13466d2adbea-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz\" (UID: \"d846102d-c04b-4cea-b451-13466d2adbea\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz"
Sep 08 13:39:32.005845 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:32.005814    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnrtb\" (UniqueName: \"kubernetes.io/projected/d846102d-c04b-4cea-b451-13466d2adbea-kube-api-access-tnrtb\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz\" (UID: \"d846102d-c04b-4cea-b451-13466d2adbea\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz"
Sep 08 13:39:32.054084 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:32.054047    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz"
Sep 08 13:39:32.334856 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:32.334828    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz"]
Sep 08 13:39:32.337523 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:39:32.337486    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd846102d_c04b_4cea_b451_13466d2adbea.slice/crio-2f6149e2bd3933eb9330886f61fc95088e950ef1d554153be9d00ca9b4298b16 WatchSource:0}: Error finding container 2f6149e2bd3933eb9330886f61fc95088e950ef1d554153be9d00ca9b4298b16: Status 404 returned error can't find the container with id 2f6149e2bd3933eb9330886f61fc95088e950ef1d554153be9d00ca9b4298b16
Sep 08 13:39:32.681113 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:32.681075    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz" event={"ID":"d846102d-c04b-4cea-b451-13466d2adbea","Type":"ContainerStarted","Data":"8a658e2dd71ec811a30053d2c24597bdecbbaae320b2fe94559646084461abde"}
Sep 08 13:39:32.681317 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:32.681141    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz" event={"ID":"d846102d-c04b-4cea-b451-13466d2adbea","Type":"ContainerStarted","Data":"2f6149e2bd3933eb9330886f61fc95088e950ef1d554153be9d00ca9b4298b16"}
Sep 08 13:39:32.708962 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:32.708912    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz" podStartSLOduration=1.7088963449999999 podStartE2EDuration="1.708896345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:39:32.707487964 +0000 UTC m=+1436.159248895" watchObservedRunningTime="2026-09-08 13:39:32.708896345 +0000 UTC m=+1436.160657275"
Sep 08 13:39:33.054742 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:33.054663    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz"
Sep 08 13:39:42.054990 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:42.054948    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz"
Sep 08 13:39:42.067572 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:42.067539    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz"
Sep 08 13:39:42.749482 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:39:42.749455    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz"
Sep 08 13:40:05.731890 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:40:05.731846    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz"]
Sep 08 13:40:05.741234 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:40:05.741185    2558 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 08 13:40:05.741234 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:40:05.741234 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:40:05.741234 ip-10-0-128-147 kubenswrapper[2558]:  > containerID="8a658e2dd71ec811a30053d2c24597bdecbbaae320b2fe94559646084461abde" cmd=["/bin/sleep","15"]
Sep 08 13:40:05.741576 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:40:05.741266    2558 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz" message="" err=<
Sep 08 13:40:05.741576 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:40:05.741576 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:40:05.741576 ip-10-0-128-147 kubenswrapper[2558]:  >
Sep 08 13:40:05.741576 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:40:05.741284    2558 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 08 13:40:05.741576 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:40:05.741576 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:40:05.741576 ip-10-0-128-147 kubenswrapper[2558]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz" podUID="d846102d-c04b-4cea-b451-13466d2adbea" containerName="main" containerID="cri-o://8a658e2dd71ec811a30053d2c24597bdecbbaae320b2fe94559646084461abde"
Sep 08 13:40:05.741576 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:40:05.741349    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz" podUID="d846102d-c04b-4cea-b451-13466d2adbea" containerName="main" containerID="cri-o://8a658e2dd71ec811a30053d2c24597bdecbbaae320b2fe94559646084461abde" gracePeriod=60
Sep 08 13:40:06.089387 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:40:06.089355    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz"
Sep 08 13:40:06.196094 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:40:06.196054    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d846102d-c04b-4cea-b451-13466d2adbea-tls-certs\") pod \"d846102d-c04b-4cea-b451-13466d2adbea\" (UID: \"d846102d-c04b-4cea-b451-13466d2adbea\") "
Sep 08 13:40:06.196094 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:40:06.196101    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d846102d-c04b-4cea-b451-13466d2adbea-home\") pod \"d846102d-c04b-4cea-b451-13466d2adbea\" (UID: \"d846102d-c04b-4cea-b451-13466d2adbea\") "
Sep 08 13:40:06.196361 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:40:06.196131    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d846102d-c04b-4cea-b451-13466d2adbea-tmp-dir\") pod \"d846102d-c04b-4cea-b451-13466d2adbea\" (UID: \"d846102d-c04b-4cea-b451-13466d2adbea\") "
Sep 08 13:40:06.196361 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:40:06.196146    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d846102d-c04b-4cea-b451-13466d2adbea-model-cache\") pod \"d846102d-c04b-4cea-b451-13466d2adbea\" (UID: \"d846102d-c04b-4cea-b451-13466d2adbea\") "
Sep 08 13:40:06.196361 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:40:06.196177    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d846102d-c04b-4cea-b451-13466d2adbea-dshm\") pod \"d846102d-c04b-4cea-b451-13466d2adbea\" (UID: \"d846102d-c04b-4cea-b451-13466d2adbea\") "
Sep 08 13:40:06.196361 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:40:06.196245    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnrtb\" (UniqueName: \"kubernetes.io/projected/d846102d-c04b-4cea-b451-13466d2adbea-kube-api-access-tnrtb\") pod \"d846102d-c04b-4cea-b451-13466d2adbea\" (UID: \"d846102d-c04b-4cea-b451-13466d2adbea\") "
Sep 08 13:40:06.196564 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:40:06.196373    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d846102d-c04b-4cea-b451-13466d2adbea-home" (OuterVolumeSpecName: "home") pod "d846102d-c04b-4cea-b451-13466d2adbea" (UID: "d846102d-c04b-4cea-b451-13466d2adbea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:40:06.196564 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:40:06.196406    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d846102d-c04b-4cea-b451-13466d2adbea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d846102d-c04b-4cea-b451-13466d2adbea" (UID: "d846102d-c04b-4cea-b451-13466d2adbea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:40:06.196564 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:40:06.196424    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d846102d-c04b-4cea-b451-13466d2adbea-model-cache" (OuterVolumeSpecName: "model-cache") pod "d846102d-c04b-4cea-b451-13466d2adbea" (UID: "d846102d-c04b-4cea-b451-13466d2adbea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:40:06.196564 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:40:06.196537    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d846102d-c04b-4cea-b451-13466d2adbea-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:40:06.196564 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:40:06.196558    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d846102d-c04b-4cea-b451-13466d2adbea-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:40:06.196818 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:40:06.196570    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d846102d-c04b-4cea-b451-13466d2adbea-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:40:06.198420 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:40:06.198396    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d846102d-c04b-4cea-b451-13466d2adbea-dshm" (OuterVolumeSpecName: "dshm") pod "d846102d-c04b-4cea-b451-13466d2adbea" (UID: "d846102d-c04b-4cea-b451-13466d2adbea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:40:06.198420 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:40:06.198408    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d846102d-c04b-4cea-b451-13466d2adbea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d846102d-c04b-4cea-b451-13466d2adbea" (UID: "d846102d-c04b-4cea-b451-13466d2adbea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:40:06.198576 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:40:06.198524    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d846102d-c04b-4cea-b451-13466d2adbea-kube-api-access-tnrtb" (OuterVolumeSpecName: "kube-api-access-tnrtb") pod "d846102d-c04b-4cea-b451-13466d2adbea" (UID: "d846102d-c04b-4cea-b451-13466d2adbea"). InnerVolumeSpecName "kube-api-access-tnrtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:40:06.297251 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:40:06.297167    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d846102d-c04b-4cea-b451-13466d2adbea-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:40:06.297251 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:40:06.297196    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d846102d-c04b-4cea-b451-13466d2adbea-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:40:06.297251 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:40:06.297205    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnrtb\" (UniqueName: \"kubernetes.io/projected/d846102d-c04b-4cea-b451-13466d2adbea-kube-api-access-tnrtb\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:40:06.861515 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:40:06.861481    2558 generic.go:356] "Generic (PLEG): container finished" podID="d846102d-c04b-4cea-b451-13466d2adbea" containerID="8a658e2dd71ec811a30053d2c24597bdecbbaae320b2fe94559646084461abde" exitCode=0
Sep 08 13:40:06.861978 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:40:06.861536    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz"
Sep 08 13:40:06.861978 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:40:06.861543    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz" event={"ID":"d846102d-c04b-4cea-b451-13466d2adbea","Type":"ContainerDied","Data":"8a658e2dd71ec811a30053d2c24597bdecbbaae320b2fe94559646084461abde"}
Sep 08 13:40:06.861978 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:40:06.861639    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz" event={"ID":"d846102d-c04b-4cea-b451-13466d2adbea","Type":"ContainerDied","Data":"2f6149e2bd3933eb9330886f61fc95088e950ef1d554153be9d00ca9b4298b16"}
Sep 08 13:40:06.861978 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:40:06.861662    2558 scope.go:117] "RemoveContainer" containerID="8a658e2dd71ec811a30053d2c24597bdecbbaae320b2fe94559646084461abde"
Sep 08 13:40:06.871281 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:40:06.871263    2558 scope.go:117] "RemoveContainer" containerID="8a658e2dd71ec811a30053d2c24597bdecbbaae320b2fe94559646084461abde"
Sep 08 13:40:06.871609 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:40:06.871575    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a658e2dd71ec811a30053d2c24597bdecbbaae320b2fe94559646084461abde\": container with ID starting with 8a658e2dd71ec811a30053d2c24597bdecbbaae320b2fe94559646084461abde not found: ID does not exist" containerID="8a658e2dd71ec811a30053d2c24597bdecbbaae320b2fe94559646084461abde"
Sep 08 13:40:06.871652 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:40:06.871618    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a658e2dd71ec811a30053d2c24597bdecbbaae320b2fe94559646084461abde"} err="failed to get container status \"8a658e2dd71ec811a30053d2c24597bdecbbaae320b2fe94559646084461abde\": rpc error: code = NotFound desc = could not find container \"8a658e2dd71ec811a30053d2c24597bdecbbaae320b2fe94559646084461abde\": container with ID starting with 8a658e2dd71ec811a30053d2c24597bdecbbaae320b2fe94559646084461abde not found: ID does not exist"
Sep 08 13:40:06.900538 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:40:06.900497    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz"]
Sep 08 13:40:06.906436 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:40:06.906404    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-h8qtz"]
Sep 08 13:40:07.052929 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:40:07.052893    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d846102d-c04b-4cea-b451-13466d2adbea" path="/var/lib/kubelet/pods/d846102d-c04b-4cea-b451-13466d2adbea/volumes"
Sep 08 13:41:01.643458 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:01.643376    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd"]
Sep 08 13:41:01.643975 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:01.643856    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d846102d-c04b-4cea-b451-13466d2adbea" containerName="main"
Sep 08 13:41:01.643975 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:01.643871    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d846102d-c04b-4cea-b451-13466d2adbea" containerName="main"
Sep 08 13:41:01.643975 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:01.643942    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d846102d-c04b-4cea-b451-13466d2adbea" containerName="main"
Sep 08 13:41:01.646830 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:01.646813    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd"
Sep 08 13:41:01.650074 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:01.650051    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-v7lbz\""
Sep 08 13:41:01.650431 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:01.650410    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 08 13:41:01.650794 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:01.650778    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 08 13:41:01.651058 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:01.651042    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 08 13:41:01.658723 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:01.658697    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd"]
Sep 08 13:41:01.787621 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:01.787554    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd\" (UID: \"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd"
Sep 08 13:41:01.787621 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:01.787625    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd\" (UID: \"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd"
Sep 08 13:41:01.787846 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:01.787690    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjcfd\" (UniqueName: \"kubernetes.io/projected/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-kube-api-access-fjcfd\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd\" (UID: \"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd"
Sep 08 13:41:01.787846 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:01.787728    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd\" (UID: \"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd"
Sep 08 13:41:01.787846 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:01.787789    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd\" (UID: \"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd"
Sep 08 13:41:01.787846 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:01.787844    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd\" (UID: \"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd"
Sep 08 13:41:01.888653 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:01.888614    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd\" (UID: \"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd"
Sep 08 13:41:01.888844 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:01.888682    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd\" (UID: \"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd"
Sep 08 13:41:01.888844 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:01.888705    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd\" (UID: \"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd"
Sep 08 13:41:01.888844 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:01.888734    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd\" (UID: \"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd"
Sep 08 13:41:01.888844 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:01.888771    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjcfd\" (UniqueName: \"kubernetes.io/projected/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-kube-api-access-fjcfd\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd\" (UID: \"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd"
Sep 08 13:41:01.889031 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:01.888840    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd\" (UID: \"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd"
Sep 08 13:41:01.889095 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:01.889072    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd\" (UID: \"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd"
Sep 08 13:41:01.889144 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:01.889124    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd\" (UID: \"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd"
Sep 08 13:41:01.889191 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:01.889141    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd\" (UID: \"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd"
Sep 08 13:41:01.891026 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:01.891007    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd\" (UID: \"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd"
Sep 08 13:41:01.891207 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:01.891187    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd\" (UID: \"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd"
Sep 08 13:41:01.899889 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:01.899827    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjcfd\" (UniqueName: \"kubernetes.io/projected/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-kube-api-access-fjcfd\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd\" (UID: \"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd"
Sep 08 13:41:01.958940 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:01.958902    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd"
Sep 08 13:41:02.259465 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:02.259436    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd"]
Sep 08 13:41:02.260994 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:41:02.260970    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda866e01e_e1a6_4fd9_87cc_67b68f8aedf0.slice/crio-9ec482e2824d9d8e980fd82e981536edf515358c2de00d820be34168f679a85f WatchSource:0}: Error finding container 9ec482e2824d9d8e980fd82e981536edf515358c2de00d820be34168f679a85f: Status 404 returned error can't find the container with id 9ec482e2824d9d8e980fd82e981536edf515358c2de00d820be34168f679a85f
Sep 08 13:41:03.132202 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:03.132161    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd" event={"ID":"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0","Type":"ContainerStarted","Data":"d5b643e53e1fecfa8685d04072fa6491e8ce8df832fa2947018361d27a68ca0a"}
Sep 08 13:41:03.132685 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:03.132220    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd" event={"ID":"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0","Type":"ContainerStarted","Data":"9ec482e2824d9d8e980fd82e981536edf515358c2de00d820be34168f679a85f"}
Sep 08 13:41:03.173072 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:03.173021    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd" podStartSLOduration=2.173005639 podStartE2EDuration="2.173005639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:41:03.170650539 +0000 UTC m=+1526.622411469" watchObservedRunningTime="2026-09-08 13:41:03.173005639 +0000 UTC m=+1526.624766569"
Sep 08 13:41:03.959190 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:03.959152    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd"
Sep 08 13:41:11.959864 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:11.959820    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd"
Sep 08 13:41:11.972586 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:11.972556    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd"
Sep 08 13:41:12.187253 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:12.187214    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd"
Sep 08 13:41:38.361019 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:38.360986    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd"]
Sep 08 13:41:38.370483 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:41:38.370440    2558 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 08 13:41:38.370483 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:41:38.370483 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:41:38.370483 ip-10-0-128-147 kubenswrapper[2558]:  > containerID="d5b643e53e1fecfa8685d04072fa6491e8ce8df832fa2947018361d27a68ca0a" cmd=["/bin/sleep","15"]
Sep 08 13:41:38.370836 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:38.370500    2558 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd" message="" err=<
Sep 08 13:41:38.370836 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:41:38.370836 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:41:38.370836 ip-10-0-128-147 kubenswrapper[2558]:  >
Sep 08 13:41:38.370836 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:41:38.370514    2558 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 08 13:41:38.370836 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:41:38.370836 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:41:38.370836 ip-10-0-128-147 kubenswrapper[2558]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd" podUID="a866e01e-e1a6-4fd9-87cc-67b68f8aedf0" containerName="main" containerID="cri-o://d5b643e53e1fecfa8685d04072fa6491e8ce8df832fa2947018361d27a68ca0a"
Sep 08 13:41:38.370836 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:38.370552    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd" podUID="a866e01e-e1a6-4fd9-87cc-67b68f8aedf0" containerName="main" containerID="cri-o://d5b643e53e1fecfa8685d04072fa6491e8ce8df832fa2947018361d27a68ca0a" gracePeriod=60
Sep 08 13:41:38.721943 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:38.721918    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd"
Sep 08 13:41:38.804496 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:38.804468    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjcfd\" (UniqueName: \"kubernetes.io/projected/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-kube-api-access-fjcfd\") pod \"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0\" (UID: \"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0\") "
Sep 08 13:41:38.804677 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:38.804517    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-model-cache\") pod \"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0\" (UID: \"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0\") "
Sep 08 13:41:38.804677 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:38.804549    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-dshm\") pod \"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0\" (UID: \"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0\") "
Sep 08 13:41:38.804677 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:38.804610    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-tls-certs\") pod \"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0\" (UID: \"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0\") "
Sep 08 13:41:38.804677 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:38.804646    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-tmp-dir\") pod \"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0\" (UID: \"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0\") "
Sep 08 13:41:38.804876 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:38.804681    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-home\") pod \"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0\" (UID: \"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0\") "
Sep 08 13:41:38.804876 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:38.804808    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-model-cache" (OuterVolumeSpecName: "model-cache") pod "a866e01e-e1a6-4fd9-87cc-67b68f8aedf0" (UID: "a866e01e-e1a6-4fd9-87cc-67b68f8aedf0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:41:38.804987 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:38.804904    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a866e01e-e1a6-4fd9-87cc-67b68f8aedf0" (UID: "a866e01e-e1a6-4fd9-87cc-67b68f8aedf0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:41:38.805043 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:38.805004    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-home" (OuterVolumeSpecName: "home") pod "a866e01e-e1a6-4fd9-87cc-67b68f8aedf0" (UID: "a866e01e-e1a6-4fd9-87cc-67b68f8aedf0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:41:38.805043 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:38.805032    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:41:38.805122 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:38.805053    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:41:38.806759 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:38.806738    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a866e01e-e1a6-4fd9-87cc-67b68f8aedf0" (UID: "a866e01e-e1a6-4fd9-87cc-67b68f8aedf0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:41:38.807172 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:38.807147    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-dshm" (OuterVolumeSpecName: "dshm") pod "a866e01e-e1a6-4fd9-87cc-67b68f8aedf0" (UID: "a866e01e-e1a6-4fd9-87cc-67b68f8aedf0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:41:38.807257 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:38.807163    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-kube-api-access-fjcfd" (OuterVolumeSpecName: "kube-api-access-fjcfd") pod "a866e01e-e1a6-4fd9-87cc-67b68f8aedf0" (UID: "a866e01e-e1a6-4fd9-87cc-67b68f8aedf0"). InnerVolumeSpecName "kube-api-access-fjcfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:41:38.906510 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:38.906424    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:41:38.906510 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:38.906452    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:41:38.906510 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:38.906461    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:41:38.906510 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:38.906470    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjcfd\" (UniqueName: \"kubernetes.io/projected/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0-kube-api-access-fjcfd\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:41:39.324012 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:39.323919    2558 generic.go:356] "Generic (PLEG): container finished" podID="a866e01e-e1a6-4fd9-87cc-67b68f8aedf0" containerID="d5b643e53e1fecfa8685d04072fa6491e8ce8df832fa2947018361d27a68ca0a" exitCode=0
Sep 08 13:41:39.324012 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:39.323982    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd"
Sep 08 13:41:39.324012 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:39.323998    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd" event={"ID":"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0","Type":"ContainerDied","Data":"d5b643e53e1fecfa8685d04072fa6491e8ce8df832fa2947018361d27a68ca0a"}
Sep 08 13:41:39.324263 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:39.324054    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd" event={"ID":"a866e01e-e1a6-4fd9-87cc-67b68f8aedf0","Type":"ContainerDied","Data":"9ec482e2824d9d8e980fd82e981536edf515358c2de00d820be34168f679a85f"}
Sep 08 13:41:39.324263 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:39.324077    2558 scope.go:117] "RemoveContainer" containerID="d5b643e53e1fecfa8685d04072fa6491e8ce8df832fa2947018361d27a68ca0a"
Sep 08 13:41:39.332835 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:39.332808    2558 scope.go:117] "RemoveContainer" containerID="d5b643e53e1fecfa8685d04072fa6491e8ce8df832fa2947018361d27a68ca0a"
Sep 08 13:41:39.333127 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:41:39.333108    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5b643e53e1fecfa8685d04072fa6491e8ce8df832fa2947018361d27a68ca0a\": container with ID starting with d5b643e53e1fecfa8685d04072fa6491e8ce8df832fa2947018361d27a68ca0a not found: ID does not exist" containerID="d5b643e53e1fecfa8685d04072fa6491e8ce8df832fa2947018361d27a68ca0a"
Sep 08 13:41:39.333180 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:39.333139    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5b643e53e1fecfa8685d04072fa6491e8ce8df832fa2947018361d27a68ca0a"} err="failed to get container status \"d5b643e53e1fecfa8685d04072fa6491e8ce8df832fa2947018361d27a68ca0a\": rpc error: code = NotFound desc = could not find container \"d5b643e53e1fecfa8685d04072fa6491e8ce8df832fa2947018361d27a68ca0a\": container with ID starting with d5b643e53e1fecfa8685d04072fa6491e8ce8df832fa2947018361d27a68ca0a not found: ID does not exist"
Sep 08 13:41:39.341770 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:39.341746    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd"]
Sep 08 13:41:39.348382 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:39.348354    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-5bvwd"]
Sep 08 13:41:41.053702 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:41:41.053667    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a866e01e-e1a6-4fd9-87cc-67b68f8aedf0" path="/var/lib/kubelet/pods/a866e01e-e1a6-4fd9-87cc-67b68f8aedf0/volumes"
Sep 08 13:42:30.043006 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:30.042925    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h"]
Sep 08 13:42:30.043431 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:30.043300    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a866e01e-e1a6-4fd9-87cc-67b68f8aedf0" containerName="main"
Sep 08 13:42:30.043431 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:30.043312    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a866e01e-e1a6-4fd9-87cc-67b68f8aedf0" containerName="main"
Sep 08 13:42:30.043431 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:30.043389    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a866e01e-e1a6-4fd9-87cc-67b68f8aedf0" containerName="main"
Sep 08 13:42:30.046020 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:30.046000    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h"
Sep 08 13:42:30.048473 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:30.048445    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 08 13:42:30.049420 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:30.049399    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 08 13:42:30.049420 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:30.049412    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-qb4bh\""
Sep 08 13:42:30.049854 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:30.049838    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 08 13:42:30.062739 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:30.062710    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h"]
Sep 08 13:42:30.126873 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:30.126841    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c4863ac-33fc-4b6e-aae9-721a95d045b2-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-wsg8h\" (UID: \"3c4863ac-33fc-4b6e-aae9-721a95d045b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h"
Sep 08 13:42:30.127070 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:30.126885    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c4863ac-33fc-4b6e-aae9-721a95d045b2-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-wsg8h\" (UID: \"3c4863ac-33fc-4b6e-aae9-721a95d045b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h"
Sep 08 13:42:30.127070 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:30.126908    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5kcc\" (UniqueName: \"kubernetes.io/projected/3c4863ac-33fc-4b6e-aae9-721a95d045b2-kube-api-access-n5kcc\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-wsg8h\" (UID: \"3c4863ac-33fc-4b6e-aae9-721a95d045b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h"
Sep 08 13:42:30.127070 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:30.126929    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c4863ac-33fc-4b6e-aae9-721a95d045b2-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-wsg8h\" (UID: \"3c4863ac-33fc-4b6e-aae9-721a95d045b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h"
Sep 08 13:42:30.127070 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:30.126978    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c4863ac-33fc-4b6e-aae9-721a95d045b2-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-wsg8h\" (UID: \"3c4863ac-33fc-4b6e-aae9-721a95d045b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h"
Sep 08 13:42:30.127070 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:30.127004    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c4863ac-33fc-4b6e-aae9-721a95d045b2-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-wsg8h\" (UID: \"3c4863ac-33fc-4b6e-aae9-721a95d045b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h"
Sep 08 13:42:30.227419 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:30.227383    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c4863ac-33fc-4b6e-aae9-721a95d045b2-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-wsg8h\" (UID: \"3c4863ac-33fc-4b6e-aae9-721a95d045b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h"
Sep 08 13:42:30.227584 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:30.227426    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c4863ac-33fc-4b6e-aae9-721a95d045b2-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-wsg8h\" (UID: \"3c4863ac-33fc-4b6e-aae9-721a95d045b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h"
Sep 08 13:42:30.227584 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:30.227545    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c4863ac-33fc-4b6e-aae9-721a95d045b2-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-wsg8h\" (UID: \"3c4863ac-33fc-4b6e-aae9-721a95d045b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h"
Sep 08 13:42:30.227693 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:30.227625    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c4863ac-33fc-4b6e-aae9-721a95d045b2-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-wsg8h\" (UID: \"3c4863ac-33fc-4b6e-aae9-721a95d045b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h"
Sep 08 13:42:30.227693 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:30.227656    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5kcc\" (UniqueName: \"kubernetes.io/projected/3c4863ac-33fc-4b6e-aae9-721a95d045b2-kube-api-access-n5kcc\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-wsg8h\" (UID: \"3c4863ac-33fc-4b6e-aae9-721a95d045b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h"
Sep 08 13:42:30.227693 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:30.227688    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c4863ac-33fc-4b6e-aae9-721a95d045b2-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-wsg8h\" (UID: \"3c4863ac-33fc-4b6e-aae9-721a95d045b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h"
Sep 08 13:42:30.227887 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:30.227859    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c4863ac-33fc-4b6e-aae9-721a95d045b2-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-wsg8h\" (UID: \"3c4863ac-33fc-4b6e-aae9-721a95d045b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h"
Sep 08 13:42:30.227954 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:30.227925    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c4863ac-33fc-4b6e-aae9-721a95d045b2-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-wsg8h\" (UID: \"3c4863ac-33fc-4b6e-aae9-721a95d045b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h"
Sep 08 13:42:30.228005 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:30.227993    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c4863ac-33fc-4b6e-aae9-721a95d045b2-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-wsg8h\" (UID: \"3c4863ac-33fc-4b6e-aae9-721a95d045b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h"
Sep 08 13:42:30.229771 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:30.229751    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c4863ac-33fc-4b6e-aae9-721a95d045b2-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-wsg8h\" (UID: \"3c4863ac-33fc-4b6e-aae9-721a95d045b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h"
Sep 08 13:42:30.230078 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:30.230055    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c4863ac-33fc-4b6e-aae9-721a95d045b2-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-wsg8h\" (UID: \"3c4863ac-33fc-4b6e-aae9-721a95d045b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h"
Sep 08 13:42:30.240357 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:30.240326    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5kcc\" (UniqueName: \"kubernetes.io/projected/3c4863ac-33fc-4b6e-aae9-721a95d045b2-kube-api-access-n5kcc\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-wsg8h\" (UID: \"3c4863ac-33fc-4b6e-aae9-721a95d045b2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h"
Sep 08 13:42:30.356798 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:30.356765    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h"
Sep 08 13:42:30.638450 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:30.637788    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h"]
Sep 08 13:42:30.640301 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:42:30.640268    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c4863ac_33fc_4b6e_aae9_721a95d045b2.slice/crio-80d146765e7e9a81795c138907aeeb7251c562823ba327d44461ede804056567 WatchSource:0}: Error finding container 80d146765e7e9a81795c138907aeeb7251c562823ba327d44461ede804056567: Status 404 returned error can't find the container with id 80d146765e7e9a81795c138907aeeb7251c562823ba327d44461ede804056567
Sep 08 13:42:31.563159 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:31.563109    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h" event={"ID":"3c4863ac-33fc-4b6e-aae9-721a95d045b2","Type":"ContainerStarted","Data":"97b140691da85ed91ab35e06ccba7f366de8530678645a28383972a71e7b8586"}
Sep 08 13:42:31.563564 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:31.563168    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h" event={"ID":"3c4863ac-33fc-4b6e-aae9-721a95d045b2","Type":"ContainerStarted","Data":"80d146765e7e9a81795c138907aeeb7251c562823ba327d44461ede804056567"}
Sep 08 13:42:31.584159 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:31.584107    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h" podStartSLOduration=1.584089976 podStartE2EDuration="1.584089976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:42:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:42:31.583733857 +0000 UTC m=+1615.035494771" watchObservedRunningTime="2026-09-08 13:42:31.584089976 +0000 UTC m=+1615.035850908"
Sep 08 13:42:32.357165 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:32.357130    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h"
Sep 08 13:42:40.357015 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:40.356981    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h"
Sep 08 13:42:40.369699 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:40.369671    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h"
Sep 08 13:42:40.619779 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:42:40.619685    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h"
Sep 08 13:43:04.317660 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:43:04.317621    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h"]
Sep 08 13:43:04.326871 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:43:04.326833    2558 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 08 13:43:04.326871 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:43:04.326871 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:43:04.326871 ip-10-0-128-147 kubenswrapper[2558]:  > containerID="97b140691da85ed91ab35e06ccba7f366de8530678645a28383972a71e7b8586" cmd=["/bin/sleep","15"]
Sep 08 13:43:04.327104 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:43:04.326888    2558 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h" message="" err=<
Sep 08 13:43:04.327104 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:43:04.327104 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:43:04.327104 ip-10-0-128-147 kubenswrapper[2558]:  >
Sep 08 13:43:04.327104 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:43:04.326902    2558 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 08 13:43:04.327104 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:43:04.327104 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:43:04.327104 ip-10-0-128-147 kubenswrapper[2558]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h" podUID="3c4863ac-33fc-4b6e-aae9-721a95d045b2" containerName="main" containerID="cri-o://97b140691da85ed91ab35e06ccba7f366de8530678645a28383972a71e7b8586"
Sep 08 13:43:04.327104 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:43:04.326937    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h" podUID="3c4863ac-33fc-4b6e-aae9-721a95d045b2" containerName="main" containerID="cri-o://97b140691da85ed91ab35e06ccba7f366de8530678645a28383972a71e7b8586" gracePeriod=60
Sep 08 13:43:04.682838 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:43:04.682811    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h"
Sep 08 13:43:04.726783 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:43:04.726751    2558 generic.go:356] "Generic (PLEG): container finished" podID="3c4863ac-33fc-4b6e-aae9-721a95d045b2" containerID="97b140691da85ed91ab35e06ccba7f366de8530678645a28383972a71e7b8586" exitCode=0
Sep 08 13:43:04.726972 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:43:04.726810    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h"
Sep 08 13:43:04.726972 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:43:04.726831    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h" event={"ID":"3c4863ac-33fc-4b6e-aae9-721a95d045b2","Type":"ContainerDied","Data":"97b140691da85ed91ab35e06ccba7f366de8530678645a28383972a71e7b8586"}
Sep 08 13:43:04.726972 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:43:04.726885    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h" event={"ID":"3c4863ac-33fc-4b6e-aae9-721a95d045b2","Type":"ContainerDied","Data":"80d146765e7e9a81795c138907aeeb7251c562823ba327d44461ede804056567"}
Sep 08 13:43:04.726972 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:43:04.726908    2558 scope.go:117] "RemoveContainer" containerID="97b140691da85ed91ab35e06ccba7f366de8530678645a28383972a71e7b8586"
Sep 08 13:43:04.735698 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:43:04.735682    2558 scope.go:117] "RemoveContainer" containerID="97b140691da85ed91ab35e06ccba7f366de8530678645a28383972a71e7b8586"
Sep 08 13:43:04.736005 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:43:04.735986    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97b140691da85ed91ab35e06ccba7f366de8530678645a28383972a71e7b8586\": container with ID starting with 97b140691da85ed91ab35e06ccba7f366de8530678645a28383972a71e7b8586 not found: ID does not exist" containerID="97b140691da85ed91ab35e06ccba7f366de8530678645a28383972a71e7b8586"
Sep 08 13:43:04.736071 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:43:04.736017    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97b140691da85ed91ab35e06ccba7f366de8530678645a28383972a71e7b8586"} err="failed to get container status \"97b140691da85ed91ab35e06ccba7f366de8530678645a28383972a71e7b8586\": rpc error: code = NotFound desc = could not find container \"97b140691da85ed91ab35e06ccba7f366de8530678645a28383972a71e7b8586\": container with ID starting with 97b140691da85ed91ab35e06ccba7f366de8530678645a28383972a71e7b8586 not found: ID does not exist"
Sep 08 13:43:04.823708 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:43:04.823665    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c4863ac-33fc-4b6e-aae9-721a95d045b2-tls-certs\") pod \"3c4863ac-33fc-4b6e-aae9-721a95d045b2\" (UID: \"3c4863ac-33fc-4b6e-aae9-721a95d045b2\") "
Sep 08 13:43:04.823708 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:43:04.823709    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5kcc\" (UniqueName: \"kubernetes.io/projected/3c4863ac-33fc-4b6e-aae9-721a95d045b2-kube-api-access-n5kcc\") pod \"3c4863ac-33fc-4b6e-aae9-721a95d045b2\" (UID: \"3c4863ac-33fc-4b6e-aae9-721a95d045b2\") "
Sep 08 13:43:04.823970 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:43:04.823732    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c4863ac-33fc-4b6e-aae9-721a95d045b2-model-cache\") pod \"3c4863ac-33fc-4b6e-aae9-721a95d045b2\" (UID: \"3c4863ac-33fc-4b6e-aae9-721a95d045b2\") "
Sep 08 13:43:04.823970 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:43:04.823756    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c4863ac-33fc-4b6e-aae9-721a95d045b2-tmp-dir\") pod \"3c4863ac-33fc-4b6e-aae9-721a95d045b2\" (UID: \"3c4863ac-33fc-4b6e-aae9-721a95d045b2\") "
Sep 08 13:43:04.823970 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:43:04.823787    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c4863ac-33fc-4b6e-aae9-721a95d045b2-home\") pod \"3c4863ac-33fc-4b6e-aae9-721a95d045b2\" (UID: \"3c4863ac-33fc-4b6e-aae9-721a95d045b2\") "
Sep 08 13:43:04.823970 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:43:04.823829    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c4863ac-33fc-4b6e-aae9-721a95d045b2-dshm\") pod \"3c4863ac-33fc-4b6e-aae9-721a95d045b2\" (UID: \"3c4863ac-33fc-4b6e-aae9-721a95d045b2\") "
Sep 08 13:43:04.824174 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:43:04.824027    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4863ac-33fc-4b6e-aae9-721a95d045b2-model-cache" (OuterVolumeSpecName: "model-cache") pod "3c4863ac-33fc-4b6e-aae9-721a95d045b2" (UID: "3c4863ac-33fc-4b6e-aae9-721a95d045b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:43:04.824174 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:43:04.824052    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4863ac-33fc-4b6e-aae9-721a95d045b2-home" (OuterVolumeSpecName: "home") pod "3c4863ac-33fc-4b6e-aae9-721a95d045b2" (UID: "3c4863ac-33fc-4b6e-aae9-721a95d045b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:43:04.824174 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:43:04.824039    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4863ac-33fc-4b6e-aae9-721a95d045b2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3c4863ac-33fc-4b6e-aae9-721a95d045b2" (UID: "3c4863ac-33fc-4b6e-aae9-721a95d045b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:43:04.826196 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:43:04.826164    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c4863ac-33fc-4b6e-aae9-721a95d045b2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3c4863ac-33fc-4b6e-aae9-721a95d045b2" (UID: "3c4863ac-33fc-4b6e-aae9-721a95d045b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:43:04.826196 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:43:04.826170    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4863ac-33fc-4b6e-aae9-721a95d045b2-dshm" (OuterVolumeSpecName: "dshm") pod "3c4863ac-33fc-4b6e-aae9-721a95d045b2" (UID: "3c4863ac-33fc-4b6e-aae9-721a95d045b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:43:04.826330 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:43:04.826235    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c4863ac-33fc-4b6e-aae9-721a95d045b2-kube-api-access-n5kcc" (OuterVolumeSpecName: "kube-api-access-n5kcc") pod "3c4863ac-33fc-4b6e-aae9-721a95d045b2" (UID: "3c4863ac-33fc-4b6e-aae9-721a95d045b2"). InnerVolumeSpecName "kube-api-access-n5kcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:43:04.924675 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:43:04.924637    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c4863ac-33fc-4b6e-aae9-721a95d045b2-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:43:04.924675 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:43:04.924669    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c4863ac-33fc-4b6e-aae9-721a95d045b2-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:43:04.924675 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:43:04.924678    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c4863ac-33fc-4b6e-aae9-721a95d045b2-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:43:04.924910 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:43:04.924690    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5kcc\" (UniqueName: \"kubernetes.io/projected/3c4863ac-33fc-4b6e-aae9-721a95d045b2-kube-api-access-n5kcc\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:43:04.924910 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:43:04.924699    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c4863ac-33fc-4b6e-aae9-721a95d045b2-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:43:04.924910 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:43:04.924708    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c4863ac-33fc-4b6e-aae9-721a95d045b2-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:43:05.054041 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:43:05.054010    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h"]
Sep 08 13:43:05.061255 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:43:05.061228    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h"]
Sep 08 13:43:05.609521 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:43:05.609472    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-wsg8h" podUID="3c4863ac-33fc-4b6e-aae9-721a95d045b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": context deadline exceeded"
Sep 08 13:43:07.053246 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:43:07.053209    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c4863ac-33fc-4b6e-aae9-721a95d045b2" path="/var/lib/kubelet/pods/3c4863ac-33fc-4b6e-aae9-721a95d045b2/volumes"
Sep 08 13:44:02.045260 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:02.045178    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"]
Sep 08 13:44:02.045657 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:02.045572    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c4863ac-33fc-4b6e-aae9-721a95d045b2" containerName="main"
Sep 08 13:44:02.045657 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:02.045585    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c4863ac-33fc-4b6e-aae9-721a95d045b2" containerName="main"
Sep 08 13:44:02.045733 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:02.045675    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c4863ac-33fc-4b6e-aae9-721a95d045b2" containerName="main"
Sep 08 13:44:02.049581 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:02.049563    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"
Sep 08 13:44:02.053507 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:02.053482    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-vhxqx\""
Sep 08 13:44:02.053839 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:02.053821    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-4sgjm\""
Sep 08 13:44:02.054223 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:02.054204    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 08 13:44:02.054269 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:02.054238    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 08 13:44:02.057779 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:02.057761    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 08 13:44:02.077494 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:02.077469    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"]
Sep 08 13:44:02.123173 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:02.123145    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/974741d5-cdb8-4201-8743-cbabb6c04dba-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65\" (UID: \"974741d5-cdb8-4201-8743-cbabb6c04dba\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"
Sep 08 13:44:02.123366 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:02.123181    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/974741d5-cdb8-4201-8743-cbabb6c04dba-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65\" (UID: \"974741d5-cdb8-4201-8743-cbabb6c04dba\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"
Sep 08 13:44:02.123366 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:02.123261    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/974741d5-cdb8-4201-8743-cbabb6c04dba-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65\" (UID: \"974741d5-cdb8-4201-8743-cbabb6c04dba\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"
Sep 08 13:44:02.123366 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:02.123292    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gq5b\" (UniqueName: \"kubernetes.io/projected/974741d5-cdb8-4201-8743-cbabb6c04dba-kube-api-access-9gq5b\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65\" (UID: \"974741d5-cdb8-4201-8743-cbabb6c04dba\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"
Sep 08 13:44:02.123366 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:02.123327    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/974741d5-cdb8-4201-8743-cbabb6c04dba-home\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65\" (UID: \"974741d5-cdb8-4201-8743-cbabb6c04dba\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"
Sep 08 13:44:02.123366 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:02.123345    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/974741d5-cdb8-4201-8743-cbabb6c04dba-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65\" (UID: \"974741d5-cdb8-4201-8743-cbabb6c04dba\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"
Sep 08 13:44:02.224152 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:02.224116    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/974741d5-cdb8-4201-8743-cbabb6c04dba-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65\" (UID: \"974741d5-cdb8-4201-8743-cbabb6c04dba\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"
Sep 08 13:44:02.224152 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:02.224150    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gq5b\" (UniqueName: \"kubernetes.io/projected/974741d5-cdb8-4201-8743-cbabb6c04dba-kube-api-access-9gq5b\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65\" (UID: \"974741d5-cdb8-4201-8743-cbabb6c04dba\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"
Sep 08 13:44:02.224396 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:02.224171    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/974741d5-cdb8-4201-8743-cbabb6c04dba-home\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65\" (UID: \"974741d5-cdb8-4201-8743-cbabb6c04dba\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"
Sep 08 13:44:02.224396 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:02.224188    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/974741d5-cdb8-4201-8743-cbabb6c04dba-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65\" (UID: \"974741d5-cdb8-4201-8743-cbabb6c04dba\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"
Sep 08 13:44:02.224396 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:02.224250    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/974741d5-cdb8-4201-8743-cbabb6c04dba-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65\" (UID: \"974741d5-cdb8-4201-8743-cbabb6c04dba\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"
Sep 08 13:44:02.224396 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:02.224272    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/974741d5-cdb8-4201-8743-cbabb6c04dba-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65\" (UID: \"974741d5-cdb8-4201-8743-cbabb6c04dba\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"
Sep 08 13:44:02.224621 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:02.224510    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/974741d5-cdb8-4201-8743-cbabb6c04dba-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65\" (UID: \"974741d5-cdb8-4201-8743-cbabb6c04dba\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"
Sep 08 13:44:02.224621 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:02.224568    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/974741d5-cdb8-4201-8743-cbabb6c04dba-home\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65\" (UID: \"974741d5-cdb8-4201-8743-cbabb6c04dba\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"
Sep 08 13:44:02.224702 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:02.224641    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/974741d5-cdb8-4201-8743-cbabb6c04dba-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65\" (UID: \"974741d5-cdb8-4201-8743-cbabb6c04dba\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"
Sep 08 13:44:02.226620 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:02.226563    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/974741d5-cdb8-4201-8743-cbabb6c04dba-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65\" (UID: \"974741d5-cdb8-4201-8743-cbabb6c04dba\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"
Sep 08 13:44:02.226906 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:02.226883    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/974741d5-cdb8-4201-8743-cbabb6c04dba-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65\" (UID: \"974741d5-cdb8-4201-8743-cbabb6c04dba\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"
Sep 08 13:44:02.233546 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:02.233523    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gq5b\" (UniqueName: \"kubernetes.io/projected/974741d5-cdb8-4201-8743-cbabb6c04dba-kube-api-access-9gq5b\") pod \"scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65\" (UID: \"974741d5-cdb8-4201-8743-cbabb6c04dba\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"
Sep 08 13:44:02.359674 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:02.359640    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"
Sep 08 13:44:02.645403 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:02.645376    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"]
Sep 08 13:44:02.647372 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:44:02.647348    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod974741d5_cdb8_4201_8743_cbabb6c04dba.slice/crio-ec11931d2644427feb9855fd49d343910fc99aaf3811ea96c2c1ff9c1067327d WatchSource:0}: Error finding container ec11931d2644427feb9855fd49d343910fc99aaf3811ea96c2c1ff9c1067327d: Status 404 returned error can't find the container with id ec11931d2644427feb9855fd49d343910fc99aaf3811ea96c2c1ff9c1067327d
Sep 08 13:44:02.649395 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:02.649379    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:44:03.004754 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:03.004653    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65" event={"ID":"974741d5-cdb8-4201-8743-cbabb6c04dba","Type":"ContainerStarted","Data":"ec11931d2644427feb9855fd49d343910fc99aaf3811ea96c2c1ff9c1067327d"}
Sep 08 13:44:05.014642 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:05.014586    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65" event={"ID":"974741d5-cdb8-4201-8743-cbabb6c04dba","Type":"ContainerStarted","Data":"68eaed2b5088f4fc680ba03cdfb63ff28fa35c9d1b407ec4e4bb2d4add654d59"}
Sep 08 13:44:05.015048 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:05.014735    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"
Sep 08 13:44:06.021587 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:06.021540    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65" event={"ID":"974741d5-cdb8-4201-8743-cbabb6c04dba","Type":"ContainerStarted","Data":"8cb87146e98ffd96bb9b4d1f46e3e62b3cdc3e14de3fef2a010b42f157a8c78f"}
Sep 08 13:44:06.046322 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:06.046270    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65" podStartSLOduration=2.27184763 podStartE2EDuration="4.046250787s" totalImagesPullingTime="1.774403157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:44:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:44:06.044777522 +0000 UTC m=+1709.496538492" watchObservedRunningTime="2026-09-08 13:44:06.046250787 +0000 UTC m=+1709.498011719"
Sep 08 13:44:06.360439 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:06.360399    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"
Sep 08 13:44:12.360814 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:12.360774    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"
Sep 08 13:44:12.373514 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:12.373483    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"
Sep 08 13:44:13.066247 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:13.066219    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"
Sep 08 13:44:14.073225 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:44:14.073195    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"
Sep 08 13:45:00.315603 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:00.315559    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"]
Sep 08 13:45:00.324651 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:45:00.324611    2558 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 08 13:45:00.324651 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:45:00.324651 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:45:00.324651 ip-10-0-128-147 kubenswrapper[2558]:  > containerID="8cb87146e98ffd96bb9b4d1f46e3e62b3cdc3e14de3fef2a010b42f157a8c78f" cmd=["/bin/sleep","15"]
Sep 08 13:45:00.324933 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:00.324668    2558 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65" message="" err=<
Sep 08 13:45:00.324933 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:45:00.324933 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:45:00.324933 ip-10-0-128-147 kubenswrapper[2558]:  >
Sep 08 13:45:00.324933 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:45:00.324683    2558 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 08 13:45:00.324933 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:45:00.324933 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:45:00.324933 ip-10-0-128-147 kubenswrapper[2558]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65" podUID="974741d5-cdb8-4201-8743-cbabb6c04dba" containerName="main" containerID="cri-o://8cb87146e98ffd96bb9b4d1f46e3e62b3cdc3e14de3fef2a010b42f157a8c78f"
Sep 08 13:45:00.324933 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:00.324739    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65" podUID="974741d5-cdb8-4201-8743-cbabb6c04dba" containerName="main" containerID="cri-o://8cb87146e98ffd96bb9b4d1f46e3e62b3cdc3e14de3fef2a010b42f157a8c78f" gracePeriod=60
Sep 08 13:45:00.541237 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:00.541201    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65" podUID="974741d5-cdb8-4201-8743-cbabb6c04dba" containerName="llm-d-routing-sidecar" containerID="cri-o://68eaed2b5088f4fc680ba03cdfb63ff28fa35c9d1b407ec4e4bb2d4add654d59" gracePeriod=60
Sep 08 13:45:00.790328 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:00.790299    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"
Sep 08 13:45:00.938918 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:00.938833    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/974741d5-cdb8-4201-8743-cbabb6c04dba-home\") pod \"974741d5-cdb8-4201-8743-cbabb6c04dba\" (UID: \"974741d5-cdb8-4201-8743-cbabb6c04dba\") "
Sep 08 13:45:00.938918 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:00.938878    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/974741d5-cdb8-4201-8743-cbabb6c04dba-tmp-dir\") pod \"974741d5-cdb8-4201-8743-cbabb6c04dba\" (UID: \"974741d5-cdb8-4201-8743-cbabb6c04dba\") "
Sep 08 13:45:00.938918 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:00.938914    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gq5b\" (UniqueName: \"kubernetes.io/projected/974741d5-cdb8-4201-8743-cbabb6c04dba-kube-api-access-9gq5b\") pod \"974741d5-cdb8-4201-8743-cbabb6c04dba\" (UID: \"974741d5-cdb8-4201-8743-cbabb6c04dba\") "
Sep 08 13:45:00.939181 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:00.938996    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/974741d5-cdb8-4201-8743-cbabb6c04dba-model-cache\") pod \"974741d5-cdb8-4201-8743-cbabb6c04dba\" (UID: \"974741d5-cdb8-4201-8743-cbabb6c04dba\") "
Sep 08 13:45:00.939181 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:00.939023    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/974741d5-cdb8-4201-8743-cbabb6c04dba-dshm\") pod \"974741d5-cdb8-4201-8743-cbabb6c04dba\" (UID: \"974741d5-cdb8-4201-8743-cbabb6c04dba\") "
Sep 08 13:45:00.939181 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:00.939078    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/974741d5-cdb8-4201-8743-cbabb6c04dba-tls-certs\") pod \"974741d5-cdb8-4201-8743-cbabb6c04dba\" (UID: \"974741d5-cdb8-4201-8743-cbabb6c04dba\") "
Sep 08 13:45:00.939181 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:00.939138    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/974741d5-cdb8-4201-8743-cbabb6c04dba-home" (OuterVolumeSpecName: "home") pod "974741d5-cdb8-4201-8743-cbabb6c04dba" (UID: "974741d5-cdb8-4201-8743-cbabb6c04dba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:45:00.939391 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:00.939191    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/974741d5-cdb8-4201-8743-cbabb6c04dba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "974741d5-cdb8-4201-8743-cbabb6c04dba" (UID: "974741d5-cdb8-4201-8743-cbabb6c04dba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:45:00.939391 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:00.939293    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/974741d5-cdb8-4201-8743-cbabb6c04dba-model-cache" (OuterVolumeSpecName: "model-cache") pod "974741d5-cdb8-4201-8743-cbabb6c04dba" (UID: "974741d5-cdb8-4201-8743-cbabb6c04dba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:45:00.939486 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:00.939393    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/974741d5-cdb8-4201-8743-cbabb6c04dba-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:45:00.939486 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:00.939410    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/974741d5-cdb8-4201-8743-cbabb6c04dba-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:45:00.939486 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:00.939425    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/974741d5-cdb8-4201-8743-cbabb6c04dba-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:45:00.941226 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:00.941200    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/974741d5-cdb8-4201-8743-cbabb6c04dba-dshm" (OuterVolumeSpecName: "dshm") pod "974741d5-cdb8-4201-8743-cbabb6c04dba" (UID: "974741d5-cdb8-4201-8743-cbabb6c04dba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:45:00.941320 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:00.941261    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/974741d5-cdb8-4201-8743-cbabb6c04dba-kube-api-access-9gq5b" (OuterVolumeSpecName: "kube-api-access-9gq5b") pod "974741d5-cdb8-4201-8743-cbabb6c04dba" (UID: "974741d5-cdb8-4201-8743-cbabb6c04dba"). InnerVolumeSpecName "kube-api-access-9gq5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:45:00.941363 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:00.941322    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/974741d5-cdb8-4201-8743-cbabb6c04dba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "974741d5-cdb8-4201-8743-cbabb6c04dba" (UID: "974741d5-cdb8-4201-8743-cbabb6c04dba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:45:01.040122 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:01.040082    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/974741d5-cdb8-4201-8743-cbabb6c04dba-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:45:01.040122 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:01.040115    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gq5b\" (UniqueName: \"kubernetes.io/projected/974741d5-cdb8-4201-8743-cbabb6c04dba-kube-api-access-9gq5b\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:45:01.040122 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:01.040124    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/974741d5-cdb8-4201-8743-cbabb6c04dba-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:45:01.291762 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:01.291659    2558 generic.go:356] "Generic (PLEG): container finished" podID="974741d5-cdb8-4201-8743-cbabb6c04dba" containerID="8cb87146e98ffd96bb9b4d1f46e3e62b3cdc3e14de3fef2a010b42f157a8c78f" exitCode=0
Sep 08 13:45:01.291762 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:01.291691    2558 generic.go:356] "Generic (PLEG): container finished" podID="974741d5-cdb8-4201-8743-cbabb6c04dba" containerID="68eaed2b5088f4fc680ba03cdfb63ff28fa35c9d1b407ec4e4bb2d4add654d59" exitCode=0
Sep 08 13:45:01.291762 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:01.291717    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"
Sep 08 13:45:01.291762 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:01.291742    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65" event={"ID":"974741d5-cdb8-4201-8743-cbabb6c04dba","Type":"ContainerDied","Data":"8cb87146e98ffd96bb9b4d1f46e3e62b3cdc3e14de3fef2a010b42f157a8c78f"}
Sep 08 13:45:01.292085 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:01.291801    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65" event={"ID":"974741d5-cdb8-4201-8743-cbabb6c04dba","Type":"ContainerDied","Data":"68eaed2b5088f4fc680ba03cdfb63ff28fa35c9d1b407ec4e4bb2d4add654d59"}
Sep 08 13:45:01.292085 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:01.291820    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65" event={"ID":"974741d5-cdb8-4201-8743-cbabb6c04dba","Type":"ContainerDied","Data":"ec11931d2644427feb9855fd49d343910fc99aaf3811ea96c2c1ff9c1067327d"}
Sep 08 13:45:01.292085 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:01.291841    2558 scope.go:117] "RemoveContainer" containerID="8cb87146e98ffd96bb9b4d1f46e3e62b3cdc3e14de3fef2a010b42f157a8c78f"
Sep 08 13:45:01.300923 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:01.300905    2558 scope.go:117] "RemoveContainer" containerID="68eaed2b5088f4fc680ba03cdfb63ff28fa35c9d1b407ec4e4bb2d4add654d59"
Sep 08 13:45:01.308560 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:01.308536    2558 scope.go:117] "RemoveContainer" containerID="8cb87146e98ffd96bb9b4d1f46e3e62b3cdc3e14de3fef2a010b42f157a8c78f"
Sep 08 13:45:01.309120 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:45:01.309096    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cb87146e98ffd96bb9b4d1f46e3e62b3cdc3e14de3fef2a010b42f157a8c78f\": container with ID starting with 8cb87146e98ffd96bb9b4d1f46e3e62b3cdc3e14de3fef2a010b42f157a8c78f not found: ID does not exist" containerID="8cb87146e98ffd96bb9b4d1f46e3e62b3cdc3e14de3fef2a010b42f157a8c78f"
Sep 08 13:45:01.309255 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:01.309130    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cb87146e98ffd96bb9b4d1f46e3e62b3cdc3e14de3fef2a010b42f157a8c78f"} err="failed to get container status \"8cb87146e98ffd96bb9b4d1f46e3e62b3cdc3e14de3fef2a010b42f157a8c78f\": rpc error: code = NotFound desc = could not find container \"8cb87146e98ffd96bb9b4d1f46e3e62b3cdc3e14de3fef2a010b42f157a8c78f\": container with ID starting with 8cb87146e98ffd96bb9b4d1f46e3e62b3cdc3e14de3fef2a010b42f157a8c78f not found: ID does not exist"
Sep 08 13:45:01.309255 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:01.309153    2558 scope.go:117] "RemoveContainer" containerID="68eaed2b5088f4fc680ba03cdfb63ff28fa35c9d1b407ec4e4bb2d4add654d59"
Sep 08 13:45:01.309445 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:45:01.309429    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68eaed2b5088f4fc680ba03cdfb63ff28fa35c9d1b407ec4e4bb2d4add654d59\": container with ID starting with 68eaed2b5088f4fc680ba03cdfb63ff28fa35c9d1b407ec4e4bb2d4add654d59 not found: ID does not exist" containerID="68eaed2b5088f4fc680ba03cdfb63ff28fa35c9d1b407ec4e4bb2d4add654d59"
Sep 08 13:45:01.309497 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:01.309449    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68eaed2b5088f4fc680ba03cdfb63ff28fa35c9d1b407ec4e4bb2d4add654d59"} err="failed to get container status \"68eaed2b5088f4fc680ba03cdfb63ff28fa35c9d1b407ec4e4bb2d4add654d59\": rpc error: code = NotFound desc = could not find container \"68eaed2b5088f4fc680ba03cdfb63ff28fa35c9d1b407ec4e4bb2d4add654d59\": container with ID starting with 68eaed2b5088f4fc680ba03cdfb63ff28fa35c9d1b407ec4e4bb2d4add654d59 not found: ID does not exist"
Sep 08 13:45:01.309497 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:01.309463    2558 scope.go:117] "RemoveContainer" containerID="8cb87146e98ffd96bb9b4d1f46e3e62b3cdc3e14de3fef2a010b42f157a8c78f"
Sep 08 13:45:01.309767 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:01.309740    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cb87146e98ffd96bb9b4d1f46e3e62b3cdc3e14de3fef2a010b42f157a8c78f"} err="failed to get container status \"8cb87146e98ffd96bb9b4d1f46e3e62b3cdc3e14de3fef2a010b42f157a8c78f\": rpc error: code = NotFound desc = could not find container \"8cb87146e98ffd96bb9b4d1f46e3e62b3cdc3e14de3fef2a010b42f157a8c78f\": container with ID starting with 8cb87146e98ffd96bb9b4d1f46e3e62b3cdc3e14de3fef2a010b42f157a8c78f not found: ID does not exist"
Sep 08 13:45:01.309767 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:01.309763    2558 scope.go:117] "RemoveContainer" containerID="68eaed2b5088f4fc680ba03cdfb63ff28fa35c9d1b407ec4e4bb2d4add654d59"
Sep 08 13:45:01.310026 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:01.310001    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68eaed2b5088f4fc680ba03cdfb63ff28fa35c9d1b407ec4e4bb2d4add654d59"} err="failed to get container status \"68eaed2b5088f4fc680ba03cdfb63ff28fa35c9d1b407ec4e4bb2d4add654d59\": rpc error: code = NotFound desc = could not find container \"68eaed2b5088f4fc680ba03cdfb63ff28fa35c9d1b407ec4e4bb2d4add654d59\": container with ID starting with 68eaed2b5088f4fc680ba03cdfb63ff28fa35c9d1b407ec4e4bb2d4add654d59 not found: ID does not exist"
Sep 08 13:45:01.310873 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:01.310852    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"]
Sep 08 13:45:01.315144 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:01.315119    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-565d5d6d6-vjw65"]
Sep 08 13:45:03.053467 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:45:03.053423    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="974741d5-cdb8-4201-8743-cbabb6c04dba" path="/var/lib/kubelet/pods/974741d5-cdb8-4201-8743-cbabb6c04dba/volumes"
Sep 08 13:46:02.267197 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.267158    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"]
Sep 08 13:46:02.267767 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.267577    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="974741d5-cdb8-4201-8743-cbabb6c04dba" containerName="main"
Sep 08 13:46:02.267767 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.267618    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="974741d5-cdb8-4201-8743-cbabb6c04dba" containerName="main"
Sep 08 13:46:02.267767 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.267650    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="974741d5-cdb8-4201-8743-cbabb6c04dba" containerName="llm-d-routing-sidecar"
Sep 08 13:46:02.267767 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.267657    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="974741d5-cdb8-4201-8743-cbabb6c04dba" containerName="llm-d-routing-sidecar"
Sep 08 13:46:02.267767 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.267717    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="974741d5-cdb8-4201-8743-cbabb6c04dba" containerName="main"
Sep 08 13:46:02.267767 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.267736    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="974741d5-cdb8-4201-8743-cbabb6c04dba" containerName="llm-d-routing-sidecar"
Sep 08 13:46:02.270803 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.270785    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"
Sep 08 13:46:02.276433 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.276409    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 08 13:46:02.276549 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.276415    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 08 13:46:02.276638 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.276438    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-9ppv2\""
Sep 08 13:46:02.276638 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.276576    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 08 13:46:02.277006 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.276988    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-t8rsf\""
Sep 08 13:46:02.290503 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.290480    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"]
Sep 08 13:46:02.464285 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.464244    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwx7d\" (UniqueName: \"kubernetes.io/projected/c4f4647a-dfe7-4d91-b104-d25fe2753432-kube-api-access-rwx7d\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl\" (UID: \"c4f4647a-dfe7-4d91-b104-d25fe2753432\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"
Sep 08 13:46:02.464461 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.464307    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4f4647a-dfe7-4d91-b104-d25fe2753432-home\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl\" (UID: \"c4f4647a-dfe7-4d91-b104-d25fe2753432\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"
Sep 08 13:46:02.464461 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.464330    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4f4647a-dfe7-4d91-b104-d25fe2753432-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl\" (UID: \"c4f4647a-dfe7-4d91-b104-d25fe2753432\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"
Sep 08 13:46:02.464461 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.464360    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4f4647a-dfe7-4d91-b104-d25fe2753432-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl\" (UID: \"c4f4647a-dfe7-4d91-b104-d25fe2753432\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"
Sep 08 13:46:02.464461 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.464376    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4f4647a-dfe7-4d91-b104-d25fe2753432-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl\" (UID: \"c4f4647a-dfe7-4d91-b104-d25fe2753432\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"
Sep 08 13:46:02.464636 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.464478    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4f4647a-dfe7-4d91-b104-d25fe2753432-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl\" (UID: \"c4f4647a-dfe7-4d91-b104-d25fe2753432\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"
Sep 08 13:46:02.565377 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.565283    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4f4647a-dfe7-4d91-b104-d25fe2753432-home\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl\" (UID: \"c4f4647a-dfe7-4d91-b104-d25fe2753432\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"
Sep 08 13:46:02.565377 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.565320    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4f4647a-dfe7-4d91-b104-d25fe2753432-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl\" (UID: \"c4f4647a-dfe7-4d91-b104-d25fe2753432\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"
Sep 08 13:46:02.565377 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.565348    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4f4647a-dfe7-4d91-b104-d25fe2753432-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl\" (UID: \"c4f4647a-dfe7-4d91-b104-d25fe2753432\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"
Sep 08 13:46:02.565377 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.565367    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4f4647a-dfe7-4d91-b104-d25fe2753432-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl\" (UID: \"c4f4647a-dfe7-4d91-b104-d25fe2753432\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"
Sep 08 13:46:02.565724 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.565405    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4f4647a-dfe7-4d91-b104-d25fe2753432-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl\" (UID: \"c4f4647a-dfe7-4d91-b104-d25fe2753432\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"
Sep 08 13:46:02.565724 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.565426    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwx7d\" (UniqueName: \"kubernetes.io/projected/c4f4647a-dfe7-4d91-b104-d25fe2753432-kube-api-access-rwx7d\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl\" (UID: \"c4f4647a-dfe7-4d91-b104-d25fe2753432\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"
Sep 08 13:46:02.565838 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.565812    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4f4647a-dfe7-4d91-b104-d25fe2753432-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl\" (UID: \"c4f4647a-dfe7-4d91-b104-d25fe2753432\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"
Sep 08 13:46:02.565947 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.565927    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4f4647a-dfe7-4d91-b104-d25fe2753432-home\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl\" (UID: \"c4f4647a-dfe7-4d91-b104-d25fe2753432\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"
Sep 08 13:46:02.566006 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.565955    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4f4647a-dfe7-4d91-b104-d25fe2753432-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl\" (UID: \"c4f4647a-dfe7-4d91-b104-d25fe2753432\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"
Sep 08 13:46:02.567781 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.567757    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4f4647a-dfe7-4d91-b104-d25fe2753432-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl\" (UID: \"c4f4647a-dfe7-4d91-b104-d25fe2753432\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"
Sep 08 13:46:02.567922 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.567904    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4f4647a-dfe7-4d91-b104-d25fe2753432-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl\" (UID: \"c4f4647a-dfe7-4d91-b104-d25fe2753432\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"
Sep 08 13:46:02.580015 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.579991    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwx7d\" (UniqueName: \"kubernetes.io/projected/c4f4647a-dfe7-4d91-b104-d25fe2753432-kube-api-access-rwx7d\") pod \"scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl\" (UID: \"c4f4647a-dfe7-4d91-b104-d25fe2753432\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"
Sep 08 13:46:02.580337 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.580305    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"
Sep 08 13:46:02.860314 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:02.858999    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"]
Sep 08 13:46:02.862072 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:46:02.862042    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4f4647a_dfe7_4d91_b104_d25fe2753432.slice/crio-8b1b7538d39651fbbeb1ae7a4d8aeddb69ffd5711f76d865b26c77c87ce69751 WatchSource:0}: Error finding container 8b1b7538d39651fbbeb1ae7a4d8aeddb69ffd5711f76d865b26c77c87ce69751: Status 404 returned error can't find the container with id 8b1b7538d39651fbbeb1ae7a4d8aeddb69ffd5711f76d865b26c77c87ce69751
Sep 08 13:46:03.590945 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:03.590908    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl" event={"ID":"c4f4647a-dfe7-4d91-b104-d25fe2753432","Type":"ContainerStarted","Data":"d8a08ce4cda777c96ff06cc2da95c8c00f528ef9c8d8f0227d67934813d0832c"}
Sep 08 13:46:03.591338 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:03.590964    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl" event={"ID":"c4f4647a-dfe7-4d91-b104-d25fe2753432","Type":"ContainerStarted","Data":"8b1b7538d39651fbbeb1ae7a4d8aeddb69ffd5711f76d865b26c77c87ce69751"}
Sep 08 13:46:03.591338 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:03.591065    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"
Sep 08 13:46:04.597640 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:04.597583    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl" event={"ID":"c4f4647a-dfe7-4d91-b104-d25fe2753432","Type":"ContainerStarted","Data":"ac6de0c983777f8ba0e2f42e4d191f07437259b351cb27ba7d615d63961bf9c9"}
Sep 08 13:46:04.618937 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:04.618883    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl" podStartSLOduration=2.618865349 podStartE2EDuration="2.618865349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:46:04.617539851 +0000 UTC m=+1828.069300826" watchObservedRunningTime="2026-09-08 13:46:04.618865349 +0000 UTC m=+1828.070626280"
Sep 08 13:46:05.581121 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:05.581080    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"
Sep 08 13:46:12.581052 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:12.580995    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"
Sep 08 13:46:12.594375 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:12.594341    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"
Sep 08 13:46:12.595526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:12.595504    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"
Sep 08 13:46:12.646635 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:12.646582    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"
Sep 08 13:46:41.315792 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:41.315760    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw"]
Sep 08 13:46:41.319224 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:41.319203    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw"
Sep 08 13:46:41.322621 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:41.322582    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"default-dockercfg-zfjv2\""
Sep 08 13:46:41.322621 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:41.322613    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v2-kserve-self-signed-certs\""
Sep 08 13:46:41.322842 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:41.322672    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"kube-root-ca.crt\""
Sep 08 13:46:41.322842 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:41.322687    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"openshift-service-ca.crt\""
Sep 08 13:46:41.330163 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:41.330139    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw"]
Sep 08 13:46:41.391799 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:41.391767    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcce4905-4a0a-4811-9b48-10e847fe212b-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-8n5vw\" (UID: \"bcce4905-4a0a-4811-9b48-10e847fe212b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw"
Sep 08 13:46:41.391980 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:41.391811    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcce4905-4a0a-4811-9b48-10e847fe212b-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-8n5vw\" (UID: \"bcce4905-4a0a-4811-9b48-10e847fe212b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw"
Sep 08 13:46:41.391980 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:41.391933    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcce4905-4a0a-4811-9b48-10e847fe212b-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-8n5vw\" (UID: \"bcce4905-4a0a-4811-9b48-10e847fe212b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw"
Sep 08 13:46:41.392099 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:41.391982    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcce4905-4a0a-4811-9b48-10e847fe212b-home\") pod \"test-div-v2-kserve-5f6fdb6b77-8n5vw\" (UID: \"bcce4905-4a0a-4811-9b48-10e847fe212b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw"
Sep 08 13:46:41.392099 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:41.392035    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcce4905-4a0a-4811-9b48-10e847fe212b-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-8n5vw\" (UID: \"bcce4905-4a0a-4811-9b48-10e847fe212b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw"
Sep 08 13:46:41.392099 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:41.392076    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjvjf\" (UniqueName: \"kubernetes.io/projected/bcce4905-4a0a-4811-9b48-10e847fe212b-kube-api-access-jjvjf\") pod \"test-div-v2-kserve-5f6fdb6b77-8n5vw\" (UID: \"bcce4905-4a0a-4811-9b48-10e847fe212b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw"
Sep 08 13:46:41.493153 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:41.493111    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcce4905-4a0a-4811-9b48-10e847fe212b-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-8n5vw\" (UID: \"bcce4905-4a0a-4811-9b48-10e847fe212b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw"
Sep 08 13:46:41.493360 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:41.493168    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcce4905-4a0a-4811-9b48-10e847fe212b-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-8n5vw\" (UID: \"bcce4905-4a0a-4811-9b48-10e847fe212b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw"
Sep 08 13:46:41.493360 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:41.493240    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcce4905-4a0a-4811-9b48-10e847fe212b-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-8n5vw\" (UID: \"bcce4905-4a0a-4811-9b48-10e847fe212b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw"
Sep 08 13:46:41.493360 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:41.493283    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcce4905-4a0a-4811-9b48-10e847fe212b-home\") pod \"test-div-v2-kserve-5f6fdb6b77-8n5vw\" (UID: \"bcce4905-4a0a-4811-9b48-10e847fe212b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw"
Sep 08 13:46:41.493360 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:41.493310    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcce4905-4a0a-4811-9b48-10e847fe212b-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-8n5vw\" (UID: \"bcce4905-4a0a-4811-9b48-10e847fe212b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw"
Sep 08 13:46:41.493360 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:41.493332    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjvjf\" (UniqueName: \"kubernetes.io/projected/bcce4905-4a0a-4811-9b48-10e847fe212b-kube-api-access-jjvjf\") pod \"test-div-v2-kserve-5f6fdb6b77-8n5vw\" (UID: \"bcce4905-4a0a-4811-9b48-10e847fe212b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw"
Sep 08 13:46:41.493684 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:41.493662    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcce4905-4a0a-4811-9b48-10e847fe212b-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-8n5vw\" (UID: \"bcce4905-4a0a-4811-9b48-10e847fe212b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw"
Sep 08 13:46:41.493782 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:41.493758    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcce4905-4a0a-4811-9b48-10e847fe212b-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-8n5vw\" (UID: \"bcce4905-4a0a-4811-9b48-10e847fe212b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw"
Sep 08 13:46:41.493829 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:41.493801    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcce4905-4a0a-4811-9b48-10e847fe212b-home\") pod \"test-div-v2-kserve-5f6fdb6b77-8n5vw\" (UID: \"bcce4905-4a0a-4811-9b48-10e847fe212b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw"
Sep 08 13:46:41.495461 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:41.495441    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcce4905-4a0a-4811-9b48-10e847fe212b-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-8n5vw\" (UID: \"bcce4905-4a0a-4811-9b48-10e847fe212b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw"
Sep 08 13:46:41.495800 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:41.495786    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcce4905-4a0a-4811-9b48-10e847fe212b-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-8n5vw\" (UID: \"bcce4905-4a0a-4811-9b48-10e847fe212b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw"
Sep 08 13:46:41.503457 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:41.503431    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjvjf\" (UniqueName: \"kubernetes.io/projected/bcce4905-4a0a-4811-9b48-10e847fe212b-kube-api-access-jjvjf\") pod \"test-div-v2-kserve-5f6fdb6b77-8n5vw\" (UID: \"bcce4905-4a0a-4811-9b48-10e847fe212b\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw"
Sep 08 13:46:41.630417 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:41.630354    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw"
Sep 08 13:46:41.906117 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:41.906078    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw"]
Sep 08 13:46:41.908457 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:46:41.908427    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcce4905_4a0a_4811_9b48_10e847fe212b.slice/crio-0fd09350c82ed90913428d82b88be99e1ea7cff61b52cf0a33a979cbe3c73f85 WatchSource:0}: Error finding container 0fd09350c82ed90913428d82b88be99e1ea7cff61b52cf0a33a979cbe3c73f85: Status 404 returned error can't find the container with id 0fd09350c82ed90913428d82b88be99e1ea7cff61b52cf0a33a979cbe3c73f85
Sep 08 13:46:42.795065 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:42.795030    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw" event={"ID":"bcce4905-4a0a-4811-9b48-10e847fe212b","Type":"ContainerStarted","Data":"0fd09350c82ed90913428d82b88be99e1ea7cff61b52cf0a33a979cbe3c73f85"}
Sep 08 13:46:43.802381 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:43.802344    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw" event={"ID":"bcce4905-4a0a-4811-9b48-10e847fe212b","Type":"ContainerStarted","Data":"933fcf71b2036d83a30c4ec99e52b85d735f3cd000f4d58547cc49b8e82bb4a2"}
Sep 08 13:46:43.823487 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:43.823432    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw" podStartSLOduration=1.6591665949999999 podStartE2EDuration="2.823414495s" totalImagesPullingTime="1.1642479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:46:43.82232819 +0000 UTC m=+1867.274089122" watchObservedRunningTime="2026-09-08 13:46:43.823414495 +0000 UTC m=+1867.275175426"
Sep 08 13:46:44.630613 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:44.630566    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw"
Sep 08 13:46:51.630553 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:51.630511    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw"
Sep 08 13:46:51.643841 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:51.643815    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw"
Sep 08 13:46:51.851960 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:46:51.851930    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw"
Sep 08 13:47:12.266994 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.266908    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"]
Sep 08 13:47:12.276206 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:47:12.276169    2558 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 08 13:47:12.276206 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:47:12.276206 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:47:12.276206 ip-10-0-128-147 kubenswrapper[2558]:  > containerID="ac6de0c983777f8ba0e2f42e4d191f07437259b351cb27ba7d615d63961bf9c9" cmd=["/bin/sleep","15"]
Sep 08 13:47:12.276456 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.276221    2558 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl" message="" err=<
Sep 08 13:47:12.276456 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:47:12.276456 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:47:12.276456 ip-10-0-128-147 kubenswrapper[2558]:  >
Sep 08 13:47:12.276456 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:47:12.276234    2558 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 08 13:47:12.276456 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:47:12.276456 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:47:12.276456 ip-10-0-128-147 kubenswrapper[2558]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl" podUID="c4f4647a-dfe7-4d91-b104-d25fe2753432" containerName="main" containerID="cri-o://ac6de0c983777f8ba0e2f42e4d191f07437259b351cb27ba7d615d63961bf9c9"
Sep 08 13:47:12.276456 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.276284    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl" podUID="c4f4647a-dfe7-4d91-b104-d25fe2753432" containerName="main" containerID="cri-o://ac6de0c983777f8ba0e2f42e4d191f07437259b351cb27ba7d615d63961bf9c9" gracePeriod=60
Sep 08 13:47:12.492531 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.492494    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl" podUID="c4f4647a-dfe7-4d91-b104-d25fe2753432" containerName="llm-d-routing-sidecar" containerID="cri-o://d8a08ce4cda777c96ff06cc2da95c8c00f528ef9c8d8f0227d67934813d0832c" gracePeriod=60
Sep 08 13:47:12.581324 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.581217    2558 prober.go:120] "Probe failed" probeType="Liveness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl" podUID="c4f4647a-dfe7-4d91-b104-d25fe2753432" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 08 13:47:12.581661 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.581631    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl" podUID="c4f4647a-dfe7-4d91-b104-d25fe2753432" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": dial tcp 10.132.0.45:8000: connect: connection refused"
Sep 08 13:47:12.636236 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.636193    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl" podUID="c4f4647a-dfe7-4d91-b104-d25fe2753432" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8001/health\": dial tcp 10.132.0.45:8001: connect: connection refused"
Sep 08 13:47:12.745363 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.745336    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"
Sep 08 13:47:12.769795 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.769760    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4f4647a-dfe7-4d91-b104-d25fe2753432-home\") pod \"c4f4647a-dfe7-4d91-b104-d25fe2753432\" (UID: \"c4f4647a-dfe7-4d91-b104-d25fe2753432\") "
Sep 08 13:47:12.769976 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.769804    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4f4647a-dfe7-4d91-b104-d25fe2753432-model-cache\") pod \"c4f4647a-dfe7-4d91-b104-d25fe2753432\" (UID: \"c4f4647a-dfe7-4d91-b104-d25fe2753432\") "
Sep 08 13:47:12.769976 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.769824    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwx7d\" (UniqueName: \"kubernetes.io/projected/c4f4647a-dfe7-4d91-b104-d25fe2753432-kube-api-access-rwx7d\") pod \"c4f4647a-dfe7-4d91-b104-d25fe2753432\" (UID: \"c4f4647a-dfe7-4d91-b104-d25fe2753432\") "
Sep 08 13:47:12.769976 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.769848    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4f4647a-dfe7-4d91-b104-d25fe2753432-tmp-dir\") pod \"c4f4647a-dfe7-4d91-b104-d25fe2753432\" (UID: \"c4f4647a-dfe7-4d91-b104-d25fe2753432\") "
Sep 08 13:47:12.769976 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.769881    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4f4647a-dfe7-4d91-b104-d25fe2753432-dshm\") pod \"c4f4647a-dfe7-4d91-b104-d25fe2753432\" (UID: \"c4f4647a-dfe7-4d91-b104-d25fe2753432\") "
Sep 08 13:47:12.769976 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.769930    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4f4647a-dfe7-4d91-b104-d25fe2753432-tls-certs\") pod \"c4f4647a-dfe7-4d91-b104-d25fe2753432\" (UID: \"c4f4647a-dfe7-4d91-b104-d25fe2753432\") "
Sep 08 13:47:12.770255 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.770106    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f4647a-dfe7-4d91-b104-d25fe2753432-model-cache" (OuterVolumeSpecName: "model-cache") pod "c4f4647a-dfe7-4d91-b104-d25fe2753432" (UID: "c4f4647a-dfe7-4d91-b104-d25fe2753432"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:47:12.770255 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.770112    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f4647a-dfe7-4d91-b104-d25fe2753432-home" (OuterVolumeSpecName: "home") pod "c4f4647a-dfe7-4d91-b104-d25fe2753432" (UID: "c4f4647a-dfe7-4d91-b104-d25fe2753432"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:47:12.770255 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.770153    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f4647a-dfe7-4d91-b104-d25fe2753432-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c4f4647a-dfe7-4d91-b104-d25fe2753432" (UID: "c4f4647a-dfe7-4d91-b104-d25fe2753432"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:47:12.770383 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.770269    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4f4647a-dfe7-4d91-b104-d25fe2753432-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:47:12.770383 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.770285    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4f4647a-dfe7-4d91-b104-d25fe2753432-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:47:12.770383 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.770294    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4f4647a-dfe7-4d91-b104-d25fe2753432-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:47:12.772467 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.772437    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4f4647a-dfe7-4d91-b104-d25fe2753432-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c4f4647a-dfe7-4d91-b104-d25fe2753432" (UID: "c4f4647a-dfe7-4d91-b104-d25fe2753432"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:47:12.772467 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.772447    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4f4647a-dfe7-4d91-b104-d25fe2753432-dshm" (OuterVolumeSpecName: "dshm") pod "c4f4647a-dfe7-4d91-b104-d25fe2753432" (UID: "c4f4647a-dfe7-4d91-b104-d25fe2753432"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:47:12.772738 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.772716    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f4647a-dfe7-4d91-b104-d25fe2753432-kube-api-access-rwx7d" (OuterVolumeSpecName: "kube-api-access-rwx7d") pod "c4f4647a-dfe7-4d91-b104-d25fe2753432" (UID: "c4f4647a-dfe7-4d91-b104-d25fe2753432"). InnerVolumeSpecName "kube-api-access-rwx7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:47:12.870893 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.870861    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwx7d\" (UniqueName: \"kubernetes.io/projected/c4f4647a-dfe7-4d91-b104-d25fe2753432-kube-api-access-rwx7d\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:47:12.870893 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.870890    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4f4647a-dfe7-4d91-b104-d25fe2753432-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:47:12.870893 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.870901    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4f4647a-dfe7-4d91-b104-d25fe2753432-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:47:12.948742 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.948694    2558 generic.go:356] "Generic (PLEG): container finished" podID="c4f4647a-dfe7-4d91-b104-d25fe2753432" containerID="ac6de0c983777f8ba0e2f42e4d191f07437259b351cb27ba7d615d63961bf9c9" exitCode=0
Sep 08 13:47:12.948742 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.948734    2558 generic.go:356] "Generic (PLEG): container finished" podID="c4f4647a-dfe7-4d91-b104-d25fe2753432" containerID="d8a08ce4cda777c96ff06cc2da95c8c00f528ef9c8d8f0227d67934813d0832c" exitCode=0
Sep 08 13:47:12.948742 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.948739    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"
Sep 08 13:47:12.949017 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.948765    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl" event={"ID":"c4f4647a-dfe7-4d91-b104-d25fe2753432","Type":"ContainerDied","Data":"ac6de0c983777f8ba0e2f42e4d191f07437259b351cb27ba7d615d63961bf9c9"}
Sep 08 13:47:12.949017 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.948846    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl" event={"ID":"c4f4647a-dfe7-4d91-b104-d25fe2753432","Type":"ContainerDied","Data":"d8a08ce4cda777c96ff06cc2da95c8c00f528ef9c8d8f0227d67934813d0832c"}
Sep 08 13:47:12.949017 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.948866    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl" event={"ID":"c4f4647a-dfe7-4d91-b104-d25fe2753432","Type":"ContainerDied","Data":"8b1b7538d39651fbbeb1ae7a4d8aeddb69ffd5711f76d865b26c77c87ce69751"}
Sep 08 13:47:12.949017 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.948887    2558 scope.go:117] "RemoveContainer" containerID="ac6de0c983777f8ba0e2f42e4d191f07437259b351cb27ba7d615d63961bf9c9"
Sep 08 13:47:12.959039 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.959002    2558 scope.go:117] "RemoveContainer" containerID="d8a08ce4cda777c96ff06cc2da95c8c00f528ef9c8d8f0227d67934813d0832c"
Sep 08 13:47:12.967068 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.967042    2558 scope.go:117] "RemoveContainer" containerID="ac6de0c983777f8ba0e2f42e4d191f07437259b351cb27ba7d615d63961bf9c9"
Sep 08 13:47:12.967434 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:47:12.967398    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac6de0c983777f8ba0e2f42e4d191f07437259b351cb27ba7d615d63961bf9c9\": container with ID starting with ac6de0c983777f8ba0e2f42e4d191f07437259b351cb27ba7d615d63961bf9c9 not found: ID does not exist" containerID="ac6de0c983777f8ba0e2f42e4d191f07437259b351cb27ba7d615d63961bf9c9"
Sep 08 13:47:12.967521 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.967437    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac6de0c983777f8ba0e2f42e4d191f07437259b351cb27ba7d615d63961bf9c9"} err="failed to get container status \"ac6de0c983777f8ba0e2f42e4d191f07437259b351cb27ba7d615d63961bf9c9\": rpc error: code = NotFound desc = could not find container \"ac6de0c983777f8ba0e2f42e4d191f07437259b351cb27ba7d615d63961bf9c9\": container with ID starting with ac6de0c983777f8ba0e2f42e4d191f07437259b351cb27ba7d615d63961bf9c9 not found: ID does not exist"
Sep 08 13:47:12.967521 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.967462    2558 scope.go:117] "RemoveContainer" containerID="d8a08ce4cda777c96ff06cc2da95c8c00f528ef9c8d8f0227d67934813d0832c"
Sep 08 13:47:12.967798 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:47:12.967778    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8a08ce4cda777c96ff06cc2da95c8c00f528ef9c8d8f0227d67934813d0832c\": container with ID starting with d8a08ce4cda777c96ff06cc2da95c8c00f528ef9c8d8f0227d67934813d0832c not found: ID does not exist" containerID="d8a08ce4cda777c96ff06cc2da95c8c00f528ef9c8d8f0227d67934813d0832c"
Sep 08 13:47:12.967929 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.967809    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a08ce4cda777c96ff06cc2da95c8c00f528ef9c8d8f0227d67934813d0832c"} err="failed to get container status \"d8a08ce4cda777c96ff06cc2da95c8c00f528ef9c8d8f0227d67934813d0832c\": rpc error: code = NotFound desc = could not find container \"d8a08ce4cda777c96ff06cc2da95c8c00f528ef9c8d8f0227d67934813d0832c\": container with ID starting with d8a08ce4cda777c96ff06cc2da95c8c00f528ef9c8d8f0227d67934813d0832c not found: ID does not exist"
Sep 08 13:47:12.967929 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.967832    2558 scope.go:117] "RemoveContainer" containerID="ac6de0c983777f8ba0e2f42e4d191f07437259b351cb27ba7d615d63961bf9c9"
Sep 08 13:47:12.968135 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.968113    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac6de0c983777f8ba0e2f42e4d191f07437259b351cb27ba7d615d63961bf9c9"} err="failed to get container status \"ac6de0c983777f8ba0e2f42e4d191f07437259b351cb27ba7d615d63961bf9c9\": rpc error: code = NotFound desc = could not find container \"ac6de0c983777f8ba0e2f42e4d191f07437259b351cb27ba7d615d63961bf9c9\": container with ID starting with ac6de0c983777f8ba0e2f42e4d191f07437259b351cb27ba7d615d63961bf9c9 not found: ID does not exist"
Sep 08 13:47:12.968194 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.968138    2558 scope.go:117] "RemoveContainer" containerID="d8a08ce4cda777c96ff06cc2da95c8c00f528ef9c8d8f0227d67934813d0832c"
Sep 08 13:47:12.968434 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.968411    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a08ce4cda777c96ff06cc2da95c8c00f528ef9c8d8f0227d67934813d0832c"} err="failed to get container status \"d8a08ce4cda777c96ff06cc2da95c8c00f528ef9c8d8f0227d67934813d0832c\": rpc error: code = NotFound desc = could not find container \"d8a08ce4cda777c96ff06cc2da95c8c00f528ef9c8d8f0227d67934813d0832c\": container with ID starting with d8a08ce4cda777c96ff06cc2da95c8c00f528ef9c8d8f0227d67934813d0832c not found: ID does not exist"
Sep 08 13:47:12.976105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.976076    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"]
Sep 08 13:47:12.980403 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:12.980382    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-b77c666fc-br2kl"]
Sep 08 13:47:13.054058 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:13.054026    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4f4647a-dfe7-4d91-b104-d25fe2753432" path="/var/lib/kubelet/pods/c4f4647a-dfe7-4d91-b104-d25fe2753432/volumes"
Sep 08 13:47:16.806836 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:16.806808    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw"]
Sep 08 13:47:28.331024 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:28.330988    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5"]
Sep 08 13:47:28.331411 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:28.331379    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4f4647a-dfe7-4d91-b104-d25fe2753432" containerName="main"
Sep 08 13:47:28.331411 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:28.331392    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4f4647a-dfe7-4d91-b104-d25fe2753432" containerName="main"
Sep 08 13:47:28.331496 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:28.331423    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4f4647a-dfe7-4d91-b104-d25fe2753432" containerName="llm-d-routing-sidecar"
Sep 08 13:47:28.331496 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:28.331429    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4f4647a-dfe7-4d91-b104-d25fe2753432" containerName="llm-d-routing-sidecar"
Sep 08 13:47:28.331567 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:28.331500    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4f4647a-dfe7-4d91-b104-d25fe2753432" containerName="llm-d-routing-sidecar"
Sep 08 13:47:28.331567 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:28.331509    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4f4647a-dfe7-4d91-b104-d25fe2753432" containerName="main"
Sep 08 13:47:28.337096 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:28.337080    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5"
Sep 08 13:47:28.340237 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:28.340219    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v1-kserve-self-signed-certs\""
Sep 08 13:47:28.341071 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:28.341049    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"openshift-service-ca.crt\""
Sep 08 13:47:28.341144 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:28.341068    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"kube-root-ca.crt\""
Sep 08 13:47:28.341396 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:28.341378    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"default-dockercfg-qwzqz\""
Sep 08 13:47:28.348155 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:28.348134    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5"]
Sep 08 13:47:28.408868 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:28.408836    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23a480c3-479b-4189-a996-a5a9b0651139-dshm\") pod \"test-div-v1-kserve-fb566d667-vdsz5\" (UID: \"23a480c3-479b-4189-a996-a5a9b0651139\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5"
Sep 08 13:47:28.409023 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:28.408881    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23a480c3-479b-4189-a996-a5a9b0651139-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-vdsz5\" (UID: \"23a480c3-479b-4189-a996-a5a9b0651139\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5"
Sep 08 13:47:28.409023 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:28.408899    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a480c3-479b-4189-a996-a5a9b0651139-tls-certs\") pod \"test-div-v1-kserve-fb566d667-vdsz5\" (UID: \"23a480c3-479b-4189-a996-a5a9b0651139\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5"
Sep 08 13:47:28.409023 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:28.408937    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23a480c3-479b-4189-a996-a5a9b0651139-home\") pod \"test-div-v1-kserve-fb566d667-vdsz5\" (UID: \"23a480c3-479b-4189-a996-a5a9b0651139\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5"
Sep 08 13:47:28.409023 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:28.408975    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a480c3-479b-4189-a996-a5a9b0651139-model-cache\") pod \"test-div-v1-kserve-fb566d667-vdsz5\" (UID: \"23a480c3-479b-4189-a996-a5a9b0651139\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5"
Sep 08 13:47:28.409023 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:28.409018    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4n74\" (UniqueName: \"kubernetes.io/projected/23a480c3-479b-4189-a996-a5a9b0651139-kube-api-access-m4n74\") pod \"test-div-v1-kserve-fb566d667-vdsz5\" (UID: \"23a480c3-479b-4189-a996-a5a9b0651139\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5"
Sep 08 13:47:28.509876 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:28.509843    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23a480c3-479b-4189-a996-a5a9b0651139-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-vdsz5\" (UID: \"23a480c3-479b-4189-a996-a5a9b0651139\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5"
Sep 08 13:47:28.509876 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:28.509877    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a480c3-479b-4189-a996-a5a9b0651139-tls-certs\") pod \"test-div-v1-kserve-fb566d667-vdsz5\" (UID: \"23a480c3-479b-4189-a996-a5a9b0651139\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5"
Sep 08 13:47:28.510104 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:28.509893    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23a480c3-479b-4189-a996-a5a9b0651139-home\") pod \"test-div-v1-kserve-fb566d667-vdsz5\" (UID: \"23a480c3-479b-4189-a996-a5a9b0651139\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5"
Sep 08 13:47:28.510104 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:28.509921    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a480c3-479b-4189-a996-a5a9b0651139-model-cache\") pod \"test-div-v1-kserve-fb566d667-vdsz5\" (UID: \"23a480c3-479b-4189-a996-a5a9b0651139\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5"
Sep 08 13:47:28.510104 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:28.509945    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4n74\" (UniqueName: \"kubernetes.io/projected/23a480c3-479b-4189-a996-a5a9b0651139-kube-api-access-m4n74\") pod \"test-div-v1-kserve-fb566d667-vdsz5\" (UID: \"23a480c3-479b-4189-a996-a5a9b0651139\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5"
Sep 08 13:47:28.510104 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:28.509986    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23a480c3-479b-4189-a996-a5a9b0651139-dshm\") pod \"test-div-v1-kserve-fb566d667-vdsz5\" (UID: \"23a480c3-479b-4189-a996-a5a9b0651139\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5"
Sep 08 13:47:28.510308 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:28.510280    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23a480c3-479b-4189-a996-a5a9b0651139-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-vdsz5\" (UID: \"23a480c3-479b-4189-a996-a5a9b0651139\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5"
Sep 08 13:47:28.510374 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:28.510330    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23a480c3-479b-4189-a996-a5a9b0651139-home\") pod \"test-div-v1-kserve-fb566d667-vdsz5\" (UID: \"23a480c3-479b-4189-a996-a5a9b0651139\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5"
Sep 08 13:47:28.510435 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:28.510392    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a480c3-479b-4189-a996-a5a9b0651139-model-cache\") pod \"test-div-v1-kserve-fb566d667-vdsz5\" (UID: \"23a480c3-479b-4189-a996-a5a9b0651139\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5"
Sep 08 13:47:28.512223 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:28.512204    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23a480c3-479b-4189-a996-a5a9b0651139-dshm\") pod \"test-div-v1-kserve-fb566d667-vdsz5\" (UID: \"23a480c3-479b-4189-a996-a5a9b0651139\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5"
Sep 08 13:47:28.512344 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:28.512328    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a480c3-479b-4189-a996-a5a9b0651139-tls-certs\") pod \"test-div-v1-kserve-fb566d667-vdsz5\" (UID: \"23a480c3-479b-4189-a996-a5a9b0651139\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5"
Sep 08 13:47:28.518562 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:28.518539    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4n74\" (UniqueName: \"kubernetes.io/projected/23a480c3-479b-4189-a996-a5a9b0651139-kube-api-access-m4n74\") pod \"test-div-v1-kserve-fb566d667-vdsz5\" (UID: \"23a480c3-479b-4189-a996-a5a9b0651139\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5"
Sep 08 13:47:28.648789 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:28.648744    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5"
Sep 08 13:47:28.932180 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:28.932141    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5"]
Sep 08 13:47:28.934499 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:47:28.934471    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a480c3_479b_4189_a996_a5a9b0651139.slice/crio-6f4a5550a1bcf2f69396f477035d7e0e2995fe63ab71a7a6aaf7652a161d4cb0 WatchSource:0}: Error finding container 6f4a5550a1bcf2f69396f477035d7e0e2995fe63ab71a7a6aaf7652a161d4cb0: Status 404 returned error can't find the container with id 6f4a5550a1bcf2f69396f477035d7e0e2995fe63ab71a7a6aaf7652a161d4cb0
Sep 08 13:47:29.027327 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:29.027294    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5" event={"ID":"23a480c3-479b-4189-a996-a5a9b0651139","Type":"ContainerStarted","Data":"c62b9f4dd6894bb69a3f527a590a77bee75a0f3ac19827f5dc8c75370346a085"}
Sep 08 13:47:29.027463 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:29.027348    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5" event={"ID":"23a480c3-479b-4189-a996-a5a9b0651139","Type":"ContainerStarted","Data":"6f4a5550a1bcf2f69396f477035d7e0e2995fe63ab71a7a6aaf7652a161d4cb0"}
Sep 08 13:47:29.049495 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:29.049421    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5" podStartSLOduration=1.049397023 podStartE2EDuration="1.049397023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:47:29.046137348 +0000 UTC m=+1912.497898291" watchObservedRunningTime="2026-09-08 13:47:29.049397023 +0000 UTC m=+1912.501157954"
Sep 08 13:47:29.649468 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:29.649425    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5"
Sep 08 13:47:31.826782 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:31.826726    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw" podUID="bcce4905-4a0a-4811-9b48-10e847fe212b" containerName="main" containerID="cri-o://933fcf71b2036d83a30c4ec99e52b85d735f3cd000f4d58547cc49b8e82bb4a2" gracePeriod=45
Sep 08 13:47:31.841091 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:31.841055    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw" podUID="bcce4905-4a0a-4811-9b48-10e847fe212b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
Sep 08 13:47:32.045155 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:32.045120    2558 generic.go:356] "Generic (PLEG): container finished" podID="bcce4905-4a0a-4811-9b48-10e847fe212b" containerID="933fcf71b2036d83a30c4ec99e52b85d735f3cd000f4d58547cc49b8e82bb4a2" exitCode=0
Sep 08 13:47:32.045323 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:32.045189    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw" event={"ID":"bcce4905-4a0a-4811-9b48-10e847fe212b","Type":"ContainerDied","Data":"933fcf71b2036d83a30c4ec99e52b85d735f3cd000f4d58547cc49b8e82bb4a2"}
Sep 08 13:47:32.076018 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:32.075991    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw"
Sep 08 13:47:32.141065 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:32.141032    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcce4905-4a0a-4811-9b48-10e847fe212b-tmp-dir\") pod \"bcce4905-4a0a-4811-9b48-10e847fe212b\" (UID: \"bcce4905-4a0a-4811-9b48-10e847fe212b\") "
Sep 08 13:47:32.141250 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:32.141077    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcce4905-4a0a-4811-9b48-10e847fe212b-tls-certs\") pod \"bcce4905-4a0a-4811-9b48-10e847fe212b\" (UID: \"bcce4905-4a0a-4811-9b48-10e847fe212b\") "
Sep 08 13:47:32.141250 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:32.141097    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjvjf\" (UniqueName: \"kubernetes.io/projected/bcce4905-4a0a-4811-9b48-10e847fe212b-kube-api-access-jjvjf\") pod \"bcce4905-4a0a-4811-9b48-10e847fe212b\" (UID: \"bcce4905-4a0a-4811-9b48-10e847fe212b\") "
Sep 08 13:47:32.141250 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:32.141127    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcce4905-4a0a-4811-9b48-10e847fe212b-model-cache\") pod \"bcce4905-4a0a-4811-9b48-10e847fe212b\" (UID: \"bcce4905-4a0a-4811-9b48-10e847fe212b\") "
Sep 08 13:47:32.141250 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:32.141172    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcce4905-4a0a-4811-9b48-10e847fe212b-dshm\") pod \"bcce4905-4a0a-4811-9b48-10e847fe212b\" (UID: \"bcce4905-4a0a-4811-9b48-10e847fe212b\") "
Sep 08 13:47:32.141250 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:32.141191    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcce4905-4a0a-4811-9b48-10e847fe212b-home\") pod \"bcce4905-4a0a-4811-9b48-10e847fe212b\" (UID: \"bcce4905-4a0a-4811-9b48-10e847fe212b\") "
Sep 08 13:47:32.141535 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:32.141350    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcce4905-4a0a-4811-9b48-10e847fe212b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bcce4905-4a0a-4811-9b48-10e847fe212b" (UID: "bcce4905-4a0a-4811-9b48-10e847fe212b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:47:32.141535 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:32.141451    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcce4905-4a0a-4811-9b48-10e847fe212b-model-cache" (OuterVolumeSpecName: "model-cache") pod "bcce4905-4a0a-4811-9b48-10e847fe212b" (UID: "bcce4905-4a0a-4811-9b48-10e847fe212b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:47:32.141535 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:32.141474    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcce4905-4a0a-4811-9b48-10e847fe212b-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:47:32.141701 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:32.141538    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcce4905-4a0a-4811-9b48-10e847fe212b-home" (OuterVolumeSpecName: "home") pod "bcce4905-4a0a-4811-9b48-10e847fe212b" (UID: "bcce4905-4a0a-4811-9b48-10e847fe212b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:47:32.143396 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:32.143371    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcce4905-4a0a-4811-9b48-10e847fe212b-kube-api-access-jjvjf" (OuterVolumeSpecName: "kube-api-access-jjvjf") pod "bcce4905-4a0a-4811-9b48-10e847fe212b" (UID: "bcce4905-4a0a-4811-9b48-10e847fe212b"). InnerVolumeSpecName "kube-api-access-jjvjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:47:32.143396 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:32.143384    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcce4905-4a0a-4811-9b48-10e847fe212b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bcce4905-4a0a-4811-9b48-10e847fe212b" (UID: "bcce4905-4a0a-4811-9b48-10e847fe212b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:47:32.143562 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:32.143389    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcce4905-4a0a-4811-9b48-10e847fe212b-dshm" (OuterVolumeSpecName: "dshm") pod "bcce4905-4a0a-4811-9b48-10e847fe212b" (UID: "bcce4905-4a0a-4811-9b48-10e847fe212b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:47:32.242837 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:32.242793    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcce4905-4a0a-4811-9b48-10e847fe212b-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:47:32.242837 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:32.242833    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjvjf\" (UniqueName: \"kubernetes.io/projected/bcce4905-4a0a-4811-9b48-10e847fe212b-kube-api-access-jjvjf\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:47:32.242837 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:32.242845    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcce4905-4a0a-4811-9b48-10e847fe212b-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:47:32.242837 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:32.242855    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcce4905-4a0a-4811-9b48-10e847fe212b-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:47:32.243102 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:32.242862    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcce4905-4a0a-4811-9b48-10e847fe212b-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:47:33.052372 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:33.052346    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw"
Sep 08 13:47:33.053216 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:33.053190    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw" event={"ID":"bcce4905-4a0a-4811-9b48-10e847fe212b","Type":"ContainerDied","Data":"0fd09350c82ed90913428d82b88be99e1ea7cff61b52cf0a33a979cbe3c73f85"}
Sep 08 13:47:33.053313 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:33.053229    2558 scope.go:117] "RemoveContainer" containerID="933fcf71b2036d83a30c4ec99e52b85d735f3cd000f4d58547cc49b8e82bb4a2"
Sep 08 13:47:33.086244 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:33.086220    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw"]
Sep 08 13:47:33.099704 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:33.099675    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-8n5vw"]
Sep 08 13:47:35.053848 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:35.053811    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcce4905-4a0a-4811-9b48-10e847fe212b" path="/var/lib/kubelet/pods/bcce4905-4a0a-4811-9b48-10e847fe212b/volumes"
Sep 08 13:47:37.832792 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:37.832759    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh"]
Sep 08 13:47:37.833165 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:37.833151    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcce4905-4a0a-4811-9b48-10e847fe212b" containerName="main"
Sep 08 13:47:37.833165 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:37.833166    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcce4905-4a0a-4811-9b48-10e847fe212b" containerName="main"
Sep 08 13:47:37.833243 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:37.833232    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcce4905-4a0a-4811-9b48-10e847fe212b" containerName="main"
Sep 08 13:47:37.837883 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:37.837859    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh"
Sep 08 13:47:37.840711 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:37.840688    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v2-kserve-self-signed-certs\""
Sep 08 13:47:37.849483 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:37.849460    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh"]
Sep 08 13:47:37.892514 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:37.892490    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4c2w\" (UniqueName: \"kubernetes.io/projected/6eaf0b84-1b6f-4976-b566-ecc1115e6775-kube-api-access-f4c2w\") pod \"test-div-v2-kserve-5f6fdb6b77-v8svh\" (UID: \"6eaf0b84-1b6f-4976-b566-ecc1115e6775\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh"
Sep 08 13:47:37.892647 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:37.892538    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eaf0b84-1b6f-4976-b566-ecc1115e6775-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-v8svh\" (UID: \"6eaf0b84-1b6f-4976-b566-ecc1115e6775\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh"
Sep 08 13:47:37.892647 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:37.892566    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eaf0b84-1b6f-4976-b566-ecc1115e6775-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-v8svh\" (UID: \"6eaf0b84-1b6f-4976-b566-ecc1115e6775\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh"
Sep 08 13:47:37.892647 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:37.892625    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6eaf0b84-1b6f-4976-b566-ecc1115e6775-home\") pod \"test-div-v2-kserve-5f6fdb6b77-v8svh\" (UID: \"6eaf0b84-1b6f-4976-b566-ecc1115e6775\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh"
Sep 08 13:47:37.892768 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:37.892652    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eaf0b84-1b6f-4976-b566-ecc1115e6775-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-v8svh\" (UID: \"6eaf0b84-1b6f-4976-b566-ecc1115e6775\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh"
Sep 08 13:47:37.892768 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:37.892675    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eaf0b84-1b6f-4976-b566-ecc1115e6775-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-v8svh\" (UID: \"6eaf0b84-1b6f-4976-b566-ecc1115e6775\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh"
Sep 08 13:47:37.993376 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:37.993341    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eaf0b84-1b6f-4976-b566-ecc1115e6775-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-v8svh\" (UID: \"6eaf0b84-1b6f-4976-b566-ecc1115e6775\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh"
Sep 08 13:47:37.993376 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:37.993379    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eaf0b84-1b6f-4976-b566-ecc1115e6775-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-v8svh\" (UID: \"6eaf0b84-1b6f-4976-b566-ecc1115e6775\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh"
Sep 08 13:47:37.993643 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:37.993408    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6eaf0b84-1b6f-4976-b566-ecc1115e6775-home\") pod \"test-div-v2-kserve-5f6fdb6b77-v8svh\" (UID: \"6eaf0b84-1b6f-4976-b566-ecc1115e6775\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh"
Sep 08 13:47:37.993643 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:37.993431    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eaf0b84-1b6f-4976-b566-ecc1115e6775-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-v8svh\" (UID: \"6eaf0b84-1b6f-4976-b566-ecc1115e6775\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh"
Sep 08 13:47:37.993643 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:37.993455    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eaf0b84-1b6f-4976-b566-ecc1115e6775-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-v8svh\" (UID: \"6eaf0b84-1b6f-4976-b566-ecc1115e6775\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh"
Sep 08 13:47:37.993643 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:37.993504    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4c2w\" (UniqueName: \"kubernetes.io/projected/6eaf0b84-1b6f-4976-b566-ecc1115e6775-kube-api-access-f4c2w\") pod \"test-div-v2-kserve-5f6fdb6b77-v8svh\" (UID: \"6eaf0b84-1b6f-4976-b566-ecc1115e6775\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh"
Sep 08 13:47:37.993924 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:37.993882    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6eaf0b84-1b6f-4976-b566-ecc1115e6775-home\") pod \"test-div-v2-kserve-5f6fdb6b77-v8svh\" (UID: \"6eaf0b84-1b6f-4976-b566-ecc1115e6775\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh"
Sep 08 13:47:37.994022 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:37.993908    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eaf0b84-1b6f-4976-b566-ecc1115e6775-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-v8svh\" (UID: \"6eaf0b84-1b6f-4976-b566-ecc1115e6775\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh"
Sep 08 13:47:37.994022 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:37.993941    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eaf0b84-1b6f-4976-b566-ecc1115e6775-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-v8svh\" (UID: \"6eaf0b84-1b6f-4976-b566-ecc1115e6775\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh"
Sep 08 13:47:37.995648 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:37.995618    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eaf0b84-1b6f-4976-b566-ecc1115e6775-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-v8svh\" (UID: \"6eaf0b84-1b6f-4976-b566-ecc1115e6775\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh"
Sep 08 13:47:37.995874 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:37.995857    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eaf0b84-1b6f-4976-b566-ecc1115e6775-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-v8svh\" (UID: \"6eaf0b84-1b6f-4976-b566-ecc1115e6775\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh"
Sep 08 13:47:38.001504 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:38.001477    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4c2w\" (UniqueName: \"kubernetes.io/projected/6eaf0b84-1b6f-4976-b566-ecc1115e6775-kube-api-access-f4c2w\") pod \"test-div-v2-kserve-5f6fdb6b77-v8svh\" (UID: \"6eaf0b84-1b6f-4976-b566-ecc1115e6775\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh"
Sep 08 13:47:38.147932 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:38.147903    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh"
Sep 08 13:47:38.428081 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:38.428047    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh"]
Sep 08 13:47:38.431643 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:47:38.431609    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eaf0b84_1b6f_4976_b566_ecc1115e6775.slice/crio-01d57680c901afe7ef15bc5e0febe580b2bca6396638b3800039e8048ff84e83 WatchSource:0}: Error finding container 01d57680c901afe7ef15bc5e0febe580b2bca6396638b3800039e8048ff84e83: Status 404 returned error can't find the container with id 01d57680c901afe7ef15bc5e0febe580b2bca6396638b3800039e8048ff84e83
Sep 08 13:47:38.649084 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:38.649050    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5"
Sep 08 13:47:38.663240 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:38.663212    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5"
Sep 08 13:47:39.083237 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:39.083196    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh" event={"ID":"6eaf0b84-1b6f-4976-b566-ecc1115e6775","Type":"ContainerStarted","Data":"bec2759114fb56039ddc267a8b216402befa975cb2b850306b6d6f0f262d5e7e"}
Sep 08 13:47:39.083776 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:39.083251    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh" event={"ID":"6eaf0b84-1b6f-4976-b566-ecc1115e6775","Type":"ContainerStarted","Data":"01d57680c901afe7ef15bc5e0febe580b2bca6396638b3800039e8048ff84e83"}
Sep 08 13:47:39.094545 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:39.094523    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5"
Sep 08 13:47:39.107611 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:39.105953    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh" podStartSLOduration=2.1059348350000002 podStartE2EDuration="2.105934835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:47:39.101373843 +0000 UTC m=+1922.553134775" watchObservedRunningTime="2026-09-08 13:47:39.105934835 +0000 UTC m=+1922.557695767"
Sep 08 13:47:39.148912 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:39.148882    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh"
Sep 08 13:47:48.148492 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:48.148459    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh"
Sep 08 13:47:48.160914 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:48.160890    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh"
Sep 08 13:47:49.143154 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:47:49.143128    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh"
Sep 08 13:48:11.054404 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:11.054371    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5"]
Sep 08 13:48:11.058309 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:11.058279    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh"]
Sep 08 13:48:18.480217 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.480183    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"]
Sep 08 13:48:18.483661 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.483639    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"
Sep 08 13:48:18.488938 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.488908    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 08 13:48:18.489747 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.489721    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 08 13:48:18.489747 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.489721    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 08 13:48:18.489945 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.489764    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:48:18.489945 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.489721    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 08 13:48:18.489945 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.489721    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-pbbdj\""
Sep 08 13:48:18.497938 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.497914    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"]
Sep 08 13:48:18.532996 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.532964    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r8qg\" (UniqueName: \"kubernetes.io/projected/c31f7bc6-8899-48f5-99bb-c8e7721528bd-kube-api-access-5r8qg\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-mrqvl\" (UID: \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"
Sep 08 13:48:18.533155 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.533005    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c31f7bc6-8899-48f5-99bb-c8e7721528bd-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-mrqvl\" (UID: \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"
Sep 08 13:48:18.533155 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.533026    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c31f7bc6-8899-48f5-99bb-c8e7721528bd-home\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-mrqvl\" (UID: \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"
Sep 08 13:48:18.533155 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.533074    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c31f7bc6-8899-48f5-99bb-c8e7721528bd-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-mrqvl\" (UID: \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"
Sep 08 13:48:18.533155 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.533122    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c31f7bc6-8899-48f5-99bb-c8e7721528bd-tls-certs\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-mrqvl\" (UID: \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"
Sep 08 13:48:18.533155 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.533155    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c31f7bc6-8899-48f5-99bb-c8e7721528bd-model-cache\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-mrqvl\" (UID: \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"
Sep 08 13:48:18.533330 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.533169    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c31f7bc6-8899-48f5-99bb-c8e7721528bd-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-mrqvl\" (UID: \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"
Sep 08 13:48:18.533330 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.533223    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c31f7bc6-8899-48f5-99bb-c8e7721528bd-dshm\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-mrqvl\" (UID: \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"
Sep 08 13:48:18.634486 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.634444    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c31f7bc6-8899-48f5-99bb-c8e7721528bd-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-mrqvl\" (UID: \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"
Sep 08 13:48:18.634486 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.634489    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c31f7bc6-8899-48f5-99bb-c8e7721528bd-home\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-mrqvl\" (UID: \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"
Sep 08 13:48:18.634703 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.634509    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c31f7bc6-8899-48f5-99bb-c8e7721528bd-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-mrqvl\" (UID: \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"
Sep 08 13:48:18.634797 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.634769    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c31f7bc6-8899-48f5-99bb-c8e7721528bd-tls-certs\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-mrqvl\" (UID: \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"
Sep 08 13:48:18.634863 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.634844    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c31f7bc6-8899-48f5-99bb-c8e7721528bd-model-cache\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-mrqvl\" (UID: \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"
Sep 08 13:48:18.634920 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.634863    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c31f7bc6-8899-48f5-99bb-c8e7721528bd-home\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-mrqvl\" (UID: \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"
Sep 08 13:48:18.634920 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.634891    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c31f7bc6-8899-48f5-99bb-c8e7721528bd-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-mrqvl\" (UID: \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"
Sep 08 13:48:18.635078 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.634912    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c31f7bc6-8899-48f5-99bb-c8e7721528bd-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-mrqvl\" (UID: \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"
Sep 08 13:48:18.635078 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.634963    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c31f7bc6-8899-48f5-99bb-c8e7721528bd-dshm\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-mrqvl\" (UID: \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"
Sep 08 13:48:18.635078 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.634992    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c31f7bc6-8899-48f5-99bb-c8e7721528bd-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-mrqvl\" (UID: \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"
Sep 08 13:48:18.635078 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.635047    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r8qg\" (UniqueName: \"kubernetes.io/projected/c31f7bc6-8899-48f5-99bb-c8e7721528bd-kube-api-access-5r8qg\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-mrqvl\" (UID: \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"
Sep 08 13:48:18.635306 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.635182    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c31f7bc6-8899-48f5-99bb-c8e7721528bd-model-cache\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-mrqvl\" (UID: \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"
Sep 08 13:48:18.635577 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.635551    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c31f7bc6-8899-48f5-99bb-c8e7721528bd-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-mrqvl\" (UID: \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"
Sep 08 13:48:18.637279 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.637258    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c31f7bc6-8899-48f5-99bb-c8e7721528bd-dshm\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-mrqvl\" (UID: \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"
Sep 08 13:48:18.637524 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.637507    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c31f7bc6-8899-48f5-99bb-c8e7721528bd-tls-certs\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-mrqvl\" (UID: \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"
Sep 08 13:48:18.648699 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.648674    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r8qg\" (UniqueName: \"kubernetes.io/projected/c31f7bc6-8899-48f5-99bb-c8e7721528bd-kube-api-access-5r8qg\") pod \"tokenizer-clean-path-test-kserve-559469c4b5-mrqvl\" (UID: \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"
Sep 08 13:48:18.796428 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.796332    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv"]
Sep 08 13:48:18.796428 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.796412    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"
Sep 08 13:48:18.800899 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.800769    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv"
Sep 08 13:48:18.823185 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.823151    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv"]
Sep 08 13:48:18.836083 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.836046    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fd1f1bf-d2e7-4909-b815-12706acd0c13-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv\" (UID: \"1fd1f1bf-d2e7-4909-b815-12706acd0c13\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv"
Sep 08 13:48:18.836223 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.836110    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fd1f1bf-d2e7-4909-b815-12706acd0c13-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv\" (UID: \"1fd1f1bf-d2e7-4909-b815-12706acd0c13\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv"
Sep 08 13:48:18.836223 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.836202    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fd1f1bf-d2e7-4909-b815-12706acd0c13-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv\" (UID: \"1fd1f1bf-d2e7-4909-b815-12706acd0c13\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv"
Sep 08 13:48:18.836319 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.836285    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd1f1bf-d2e7-4909-b815-12706acd0c13-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv\" (UID: \"1fd1f1bf-d2e7-4909-b815-12706acd0c13\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv"
Sep 08 13:48:18.936922 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.936888    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd1f1bf-d2e7-4909-b815-12706acd0c13-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv\" (UID: \"1fd1f1bf-d2e7-4909-b815-12706acd0c13\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv"
Sep 08 13:48:18.937125 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.936943    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fd1f1bf-d2e7-4909-b815-12706acd0c13-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv\" (UID: \"1fd1f1bf-d2e7-4909-b815-12706acd0c13\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv"
Sep 08 13:48:18.937125 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.936970    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fd1f1bf-d2e7-4909-b815-12706acd0c13-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv\" (UID: \"1fd1f1bf-d2e7-4909-b815-12706acd0c13\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv"
Sep 08 13:48:18.937125 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.936992    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fd1f1bf-d2e7-4909-b815-12706acd0c13-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv\" (UID: \"1fd1f1bf-d2e7-4909-b815-12706acd0c13\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv"
Sep 08 13:48:18.937333 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.937314    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fd1f1bf-d2e7-4909-b815-12706acd0c13-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv\" (UID: \"1fd1f1bf-d2e7-4909-b815-12706acd0c13\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv"
Sep 08 13:48:18.937458 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.937435    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fd1f1bf-d2e7-4909-b815-12706acd0c13-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv\" (UID: \"1fd1f1bf-d2e7-4909-b815-12706acd0c13\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv"
Sep 08 13:48:18.937717 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.937699    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fd1f1bf-d2e7-4909-b815-12706acd0c13-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv\" (UID: \"1fd1f1bf-d2e7-4909-b815-12706acd0c13\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv"
Sep 08 13:48:18.939557 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:18.939537    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd1f1bf-d2e7-4909-b815-12706acd0c13-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv\" (UID: \"1fd1f1bf-d2e7-4909-b815-12706acd0c13\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv"
Sep 08 13:48:19.083127 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:19.083102    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"]
Sep 08 13:48:19.085054 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:48:19.085030    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc31f7bc6_8899_48f5_99bb_c8e7721528bd.slice/crio-ceeacb5012d0146d281b9096b8b57c8bd137cc34c6a5d84b7492fb1434c3b0cf WatchSource:0}: Error finding container ceeacb5012d0146d281b9096b8b57c8bd137cc34c6a5d84b7492fb1434c3b0cf: Status 404 returned error can't find the container with id ceeacb5012d0146d281b9096b8b57c8bd137cc34c6a5d84b7492fb1434c3b0cf
Sep 08 13:48:19.136254 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:19.136228    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv"
Sep 08 13:48:19.276841 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:19.276806    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl" event={"ID":"c31f7bc6-8899-48f5-99bb-c8e7721528bd","Type":"ContainerStarted","Data":"a940a597561af89f2cfff3e27781701bcdac17572150b4802fbcb1ddba68aff0"}
Sep 08 13:48:19.277011 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:19.276866    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl" event={"ID":"c31f7bc6-8899-48f5-99bb-c8e7721528bd","Type":"ContainerStarted","Data":"ceeacb5012d0146d281b9096b8b57c8bd137cc34c6a5d84b7492fb1434c3b0cf"}
Sep 08 13:48:19.413619 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:19.413576    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv"]
Sep 08 13:48:19.416937 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:48:19.416906    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fd1f1bf_d2e7_4909_b815_12706acd0c13.slice/crio-e67be1be529d1bd7f3158d472440f7314df15f3f00027bb992a17c3a2f68f128 WatchSource:0}: Error finding container e67be1be529d1bd7f3158d472440f7314df15f3f00027bb992a17c3a2f68f128: Status 404 returned error can't find the container with id e67be1be529d1bd7f3158d472440f7314df15f3f00027bb992a17c3a2f68f128
Sep 08 13:48:20.283480 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:20.283440    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv" event={"ID":"1fd1f1bf-d2e7-4909-b815-12706acd0c13","Type":"ContainerStarted","Data":"5046ee26fc699bdb4867bc39d03df4d367750fa5152586d03898f694fdff6d1c"}
Sep 08 13:48:20.283955 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:20.283510    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv" event={"ID":"1fd1f1bf-d2e7-4909-b815-12706acd0c13","Type":"ContainerStarted","Data":"e67be1be529d1bd7f3158d472440f7314df15f3f00027bb992a17c3a2f68f128"}
Sep 08 13:48:22.294912 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:22.294872    2558 generic.go:356] "Generic (PLEG): container finished" podID="1fd1f1bf-d2e7-4909-b815-12706acd0c13" containerID="5046ee26fc699bdb4867bc39d03df4d367750fa5152586d03898f694fdff6d1c" exitCode=0
Sep 08 13:48:22.295357 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:22.294948    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv" event={"ID":"1fd1f1bf-d2e7-4909-b815-12706acd0c13","Type":"ContainerDied","Data":"5046ee26fc699bdb4867bc39d03df4d367750fa5152586d03898f694fdff6d1c"}
Sep 08 13:48:22.296750 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:22.296672    2558 generic.go:356] "Generic (PLEG): container finished" podID="c31f7bc6-8899-48f5-99bb-c8e7721528bd" containerID="a940a597561af89f2cfff3e27781701bcdac17572150b4802fbcb1ddba68aff0" exitCode=0
Sep 08 13:48:22.296836 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:22.296757    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl" event={"ID":"c31f7bc6-8899-48f5-99bb-c8e7721528bd","Type":"ContainerDied","Data":"a940a597561af89f2cfff3e27781701bcdac17572150b4802fbcb1ddba68aff0"}
Sep 08 13:48:23.304575 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:23.304537    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl" event={"ID":"c31f7bc6-8899-48f5-99bb-c8e7721528bd","Type":"ContainerStarted","Data":"58f3c4946501d1bf1eee79530c4f47e76f4d964e30d2731f5ec59bc93ce93398"}
Sep 08 13:48:23.345178 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:23.345127    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl" podStartSLOduration=2.734694865 podStartE2EDuration="5.345111023s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.610416158s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:48:23.3416684 +0000 UTC m=+1966.793429330" watchObservedRunningTime="2026-09-08 13:48:23.345111023 +0000 UTC m=+1966.796871954"
Sep 08 13:48:23.797423 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:23.797383    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"
Sep 08 13:48:26.074932 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.074884    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5" podUID="23a480c3-479b-4189-a996-a5a9b0651139" containerName="main" containerID="cri-o://c62b9f4dd6894bb69a3f527a590a77bee75a0f3ac19827f5dc8c75370346a085" gracePeriod=45
Sep 08 13:48:26.077830 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.077650    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh" podUID="6eaf0b84-1b6f-4976-b566-ecc1115e6775" containerName="main" containerID="cri-o://bec2759114fb56039ddc267a8b216402befa975cb2b850306b6d6f0f262d5e7e" gracePeriod=45
Sep 08 13:48:26.083784 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.083745    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5" podUID="23a480c3-479b-4189-a996-a5a9b0651139" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": dial tcp 10.132.0.47:8000: connect: connection refused"
Sep 08 13:48:26.132469 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.132417    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh" podUID="6eaf0b84-1b6f-4976-b566-ecc1115e6775" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": dial tcp 10.132.0.48:8000: connect: connection refused"
Sep 08 13:48:26.329964 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.329732    2558 generic.go:356] "Generic (PLEG): container finished" podID="23a480c3-479b-4189-a996-a5a9b0651139" containerID="c62b9f4dd6894bb69a3f527a590a77bee75a0f3ac19827f5dc8c75370346a085" exitCode=0
Sep 08 13:48:26.329964 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.329826    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5" event={"ID":"23a480c3-479b-4189-a996-a5a9b0651139","Type":"ContainerDied","Data":"c62b9f4dd6894bb69a3f527a590a77bee75a0f3ac19827f5dc8c75370346a085"}
Sep 08 13:48:26.332876 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.332765    2558 generic.go:356] "Generic (PLEG): container finished" podID="6eaf0b84-1b6f-4976-b566-ecc1115e6775" containerID="bec2759114fb56039ddc267a8b216402befa975cb2b850306b6d6f0f262d5e7e" exitCode=0
Sep 08 13:48:26.332876 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.332808    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh" event={"ID":"6eaf0b84-1b6f-4976-b566-ecc1115e6775","Type":"ContainerDied","Data":"bec2759114fb56039ddc267a8b216402befa975cb2b850306b6d6f0f262d5e7e"}
Sep 08 13:48:26.374547 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.374512    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5"
Sep 08 13:48:26.404930 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.404887    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh"
Sep 08 13:48:26.412760 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.412729    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23a480c3-479b-4189-a996-a5a9b0651139-dshm\") pod \"23a480c3-479b-4189-a996-a5a9b0651139\" (UID: \"23a480c3-479b-4189-a996-a5a9b0651139\") "
Sep 08 13:48:26.412925 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.412769    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23a480c3-479b-4189-a996-a5a9b0651139-tmp-dir\") pod \"23a480c3-479b-4189-a996-a5a9b0651139\" (UID: \"23a480c3-479b-4189-a996-a5a9b0651139\") "
Sep 08 13:48:26.412925 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.412849    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23a480c3-479b-4189-a996-a5a9b0651139-home\") pod \"23a480c3-479b-4189-a996-a5a9b0651139\" (UID: \"23a480c3-479b-4189-a996-a5a9b0651139\") "
Sep 08 13:48:26.412925 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.412885    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4n74\" (UniqueName: \"kubernetes.io/projected/23a480c3-479b-4189-a996-a5a9b0651139-kube-api-access-m4n74\") pod \"23a480c3-479b-4189-a996-a5a9b0651139\" (UID: \"23a480c3-479b-4189-a996-a5a9b0651139\") "
Sep 08 13:48:26.413092 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.412938    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a480c3-479b-4189-a996-a5a9b0651139-tls-certs\") pod \"23a480c3-479b-4189-a996-a5a9b0651139\" (UID: \"23a480c3-479b-4189-a996-a5a9b0651139\") "
Sep 08 13:48:26.413092 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.412995    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a480c3-479b-4189-a996-a5a9b0651139-model-cache\") pod \"23a480c3-479b-4189-a996-a5a9b0651139\" (UID: \"23a480c3-479b-4189-a996-a5a9b0651139\") "
Sep 08 13:48:26.413302 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.413272    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a480c3-479b-4189-a996-a5a9b0651139-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "23a480c3-479b-4189-a996-a5a9b0651139" (UID: "23a480c3-479b-4189-a996-a5a9b0651139"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:48:26.413512 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.413489    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a480c3-479b-4189-a996-a5a9b0651139-model-cache" (OuterVolumeSpecName: "model-cache") pod "23a480c3-479b-4189-a996-a5a9b0651139" (UID: "23a480c3-479b-4189-a996-a5a9b0651139"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:48:26.413512 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.413503    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a480c3-479b-4189-a996-a5a9b0651139-home" (OuterVolumeSpecName: "home") pod "23a480c3-479b-4189-a996-a5a9b0651139" (UID: "23a480c3-479b-4189-a996-a5a9b0651139"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:48:26.416230 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.416162    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a480c3-479b-4189-a996-a5a9b0651139-dshm" (OuterVolumeSpecName: "dshm") pod "23a480c3-479b-4189-a996-a5a9b0651139" (UID: "23a480c3-479b-4189-a996-a5a9b0651139"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:48:26.416548 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.416444    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a480c3-479b-4189-a996-a5a9b0651139-kube-api-access-m4n74" (OuterVolumeSpecName: "kube-api-access-m4n74") pod "23a480c3-479b-4189-a996-a5a9b0651139" (UID: "23a480c3-479b-4189-a996-a5a9b0651139"). InnerVolumeSpecName "kube-api-access-m4n74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:48:26.416548 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.416526    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23a480c3-479b-4189-a996-a5a9b0651139-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "23a480c3-479b-4189-a996-a5a9b0651139" (UID: "23a480c3-479b-4189-a996-a5a9b0651139"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:48:26.513764 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.513693    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eaf0b84-1b6f-4976-b566-ecc1115e6775-dshm\") pod \"6eaf0b84-1b6f-4976-b566-ecc1115e6775\" (UID: \"6eaf0b84-1b6f-4976-b566-ecc1115e6775\") "
Sep 08 13:48:26.513764 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.513764    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eaf0b84-1b6f-4976-b566-ecc1115e6775-model-cache\") pod \"6eaf0b84-1b6f-4976-b566-ecc1115e6775\" (UID: \"6eaf0b84-1b6f-4976-b566-ecc1115e6775\") "
Sep 08 13:48:26.514011 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.513849    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6eaf0b84-1b6f-4976-b566-ecc1115e6775-home\") pod \"6eaf0b84-1b6f-4976-b566-ecc1115e6775\" (UID: \"6eaf0b84-1b6f-4976-b566-ecc1115e6775\") "
Sep 08 13:48:26.514011 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.513901    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4c2w\" (UniqueName: \"kubernetes.io/projected/6eaf0b84-1b6f-4976-b566-ecc1115e6775-kube-api-access-f4c2w\") pod \"6eaf0b84-1b6f-4976-b566-ecc1115e6775\" (UID: \"6eaf0b84-1b6f-4976-b566-ecc1115e6775\") "
Sep 08 13:48:26.514011 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.513949    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eaf0b84-1b6f-4976-b566-ecc1115e6775-tmp-dir\") pod \"6eaf0b84-1b6f-4976-b566-ecc1115e6775\" (UID: \"6eaf0b84-1b6f-4976-b566-ecc1115e6775\") "
Sep 08 13:48:26.514011 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.513976    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eaf0b84-1b6f-4976-b566-ecc1115e6775-tls-certs\") pod \"6eaf0b84-1b6f-4976-b566-ecc1115e6775\" (UID: \"6eaf0b84-1b6f-4976-b566-ecc1115e6775\") "
Sep 08 13:48:26.514620 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.514269    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a480c3-479b-4189-a996-a5a9b0651139-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:48:26.514620 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.514278    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eaf0b84-1b6f-4976-b566-ecc1115e6775-model-cache" (OuterVolumeSpecName: "model-cache") pod "6eaf0b84-1b6f-4976-b566-ecc1115e6775" (UID: "6eaf0b84-1b6f-4976-b566-ecc1115e6775"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:48:26.514620 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.514294    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a480c3-479b-4189-a996-a5a9b0651139-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:48:26.514620 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.514364    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23a480c3-479b-4189-a996-a5a9b0651139-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:48:26.514620 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.514469    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eaf0b84-1b6f-4976-b566-ecc1115e6775-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6eaf0b84-1b6f-4976-b566-ecc1115e6775" (UID: "6eaf0b84-1b6f-4976-b566-ecc1115e6775"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:48:26.514620 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.514480    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23a480c3-479b-4189-a996-a5a9b0651139-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:48:26.514620 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.514548    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23a480c3-479b-4189-a996-a5a9b0651139-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:48:26.514620 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.514567    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4n74\" (UniqueName: \"kubernetes.io/projected/23a480c3-479b-4189-a996-a5a9b0651139-kube-api-access-m4n74\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:48:26.515090 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.514786    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eaf0b84-1b6f-4976-b566-ecc1115e6775-home" (OuterVolumeSpecName: "home") pod "6eaf0b84-1b6f-4976-b566-ecc1115e6775" (UID: "6eaf0b84-1b6f-4976-b566-ecc1115e6775"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:48:26.518183 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.518145    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eaf0b84-1b6f-4976-b566-ecc1115e6775-dshm" (OuterVolumeSpecName: "dshm") pod "6eaf0b84-1b6f-4976-b566-ecc1115e6775" (UID: "6eaf0b84-1b6f-4976-b566-ecc1115e6775"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:48:26.518311 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.518181    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eaf0b84-1b6f-4976-b566-ecc1115e6775-kube-api-access-f4c2w" (OuterVolumeSpecName: "kube-api-access-f4c2w") pod "6eaf0b84-1b6f-4976-b566-ecc1115e6775" (UID: "6eaf0b84-1b6f-4976-b566-ecc1115e6775"). InnerVolumeSpecName "kube-api-access-f4c2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:48:26.518908 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.518874    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6eaf0b84-1b6f-4976-b566-ecc1115e6775-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6eaf0b84-1b6f-4976-b566-ecc1115e6775" (UID: "6eaf0b84-1b6f-4976-b566-ecc1115e6775"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:48:26.615389 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.615352    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eaf0b84-1b6f-4976-b566-ecc1115e6775-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:48:26.615389 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.615386    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eaf0b84-1b6f-4976-b566-ecc1115e6775-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:48:26.615389 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.615398    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eaf0b84-1b6f-4976-b566-ecc1115e6775-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:48:26.615743 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.615409    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eaf0b84-1b6f-4976-b566-ecc1115e6775-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:48:26.615743 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.615420    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6eaf0b84-1b6f-4976-b566-ecc1115e6775-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:48:26.615743 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:26.615432    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4c2w\" (UniqueName: \"kubernetes.io/projected/6eaf0b84-1b6f-4976-b566-ecc1115e6775-kube-api-access-f4c2w\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:48:27.342681 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:27.342580    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5"
Sep 08 13:48:27.343140 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:27.342687    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5" event={"ID":"23a480c3-479b-4189-a996-a5a9b0651139","Type":"ContainerDied","Data":"6f4a5550a1bcf2f69396f477035d7e0e2995fe63ab71a7a6aaf7652a161d4cb0"}
Sep 08 13:48:27.343140 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:27.342760    2558 scope.go:117] "RemoveContainer" containerID="c62b9f4dd6894bb69a3f527a590a77bee75a0f3ac19827f5dc8c75370346a085"
Sep 08 13:48:27.344727 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:27.344608    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh" event={"ID":"6eaf0b84-1b6f-4976-b566-ecc1115e6775","Type":"ContainerDied","Data":"01d57680c901afe7ef15bc5e0febe580b2bca6396638b3800039e8048ff84e83"}
Sep 08 13:48:27.344727 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:27.344622    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh"
Sep 08 13:48:27.355515 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:27.355473    2558 scope.go:117] "RemoveContainer" containerID="bec2759114fb56039ddc267a8b216402befa975cb2b850306b6d6f0f262d5e7e"
Sep 08 13:48:27.371519 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:27.371479    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5"]
Sep 08 13:48:27.374180 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:27.374140    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-vdsz5"]
Sep 08 13:48:27.386725 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:27.386697    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh"]
Sep 08 13:48:27.391609 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:27.391566    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-v8svh"]
Sep 08 13:48:28.797432 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:28.797189    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"
Sep 08 13:48:28.816478 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:28.816436    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"
Sep 08 13:48:29.056760 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:29.056671    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23a480c3-479b-4189-a996-a5a9b0651139" path="/var/lib/kubelet/pods/23a480c3-479b-4189-a996-a5a9b0651139/volumes"
Sep 08 13:48:29.057341 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:29.057255    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6eaf0b84-1b6f-4976-b566-ecc1115e6775" path="/var/lib/kubelet/pods/6eaf0b84-1b6f-4976-b566-ecc1115e6775/volumes"
Sep 08 13:48:29.377259 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:48:29.377227    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"
Sep 08 13:49:09.604726 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:09.603465    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv" event={"ID":"1fd1f1bf-d2e7-4909-b815-12706acd0c13","Type":"ContainerStarted","Data":"a7069bf0f1f33f9a96e6676f1634ea06eee1bbe94c39cc698f47c7058471401f"}
Sep 08 13:49:09.604726 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:09.604556    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv"
Sep 08 13:49:09.625291 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:09.625237    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv" podStartSLOduration=2.223834425 podStartE2EDuration="51.625220072s" totalImagesPullingTime="46.832337824s" totalInitContainerRuntime="2.569047823s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:48:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:49:09.623736406 +0000 UTC m=+2013.075497340" watchObservedRunningTime="2026-09-08 13:49:09.625220072 +0000 UTC m=+2013.076980994"
Sep 08 13:49:40.620351 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:40.620309    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv"
Sep 08 13:49:42.356618 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:42.356571    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv_1fd1f1bf-d2e7-4909-b815-12706acd0c13/main/0.log"
Sep 08 13:49:42.625015 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:42.624925    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"]
Sep 08 13:49:42.648285 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:42.648251    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv"]
Sep 08 13:49:42.648537 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:42.648510    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv" podUID="1fd1f1bf-d2e7-4909-b815-12706acd0c13" containerName="main" containerID="cri-o://a7069bf0f1f33f9a96e6676f1634ea06eee1bbe94c39cc698f47c7058471401f" gracePeriod=30
Sep 08 13:49:42.775009 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:49:42.774974    2558 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 08 13:49:42.775009 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:49:42.775009 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:49:42.775009 ip-10-0-128-147 kubenswrapper[2558]:  > containerID="58f3c4946501d1bf1eee79530c4f47e76f4d964e30d2731f5ec59bc93ce93398" cmd=["/bin/sleep","15"]
Sep 08 13:49:42.775321 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:42.775029    2558 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl" message="" err=<
Sep 08 13:49:42.775321 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:49:42.775321 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:49:42.775321 ip-10-0-128-147 kubenswrapper[2558]:  >
Sep 08 13:49:42.775321 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:49:42.775045    2558 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 08 13:49:42.775321 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:49:42.775321 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:49:42.775321 ip-10-0-128-147 kubenswrapper[2558]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl" podUID="c31f7bc6-8899-48f5-99bb-c8e7721528bd" containerName="main" containerID="cri-o://58f3c4946501d1bf1eee79530c4f47e76f4d964e30d2731f5ec59bc93ce93398"
Sep 08 13:49:42.775321 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:42.775087    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl" podUID="c31f7bc6-8899-48f5-99bb-c8e7721528bd" containerName="main" containerID="cri-o://58f3c4946501d1bf1eee79530c4f47e76f4d964e30d2731f5ec59bc93ce93398" gracePeriod=60
Sep 08 13:49:43.126010 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.125781    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"
Sep 08 13:49:43.151136 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.151075    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c31f7bc6-8899-48f5-99bb-c8e7721528bd-tls-certs\") pod \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\" (UID: \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\") "
Sep 08 13:49:43.151136 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.151123    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c31f7bc6-8899-48f5-99bb-c8e7721528bd-dshm\") pod \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\" (UID: \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\") "
Sep 08 13:49:43.151345 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.151155    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c31f7bc6-8899-48f5-99bb-c8e7721528bd-tmp-dir\") pod \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\" (UID: \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\") "
Sep 08 13:49:43.151345 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.151208    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c31f7bc6-8899-48f5-99bb-c8e7721528bd-home\") pod \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\" (UID: \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\") "
Sep 08 13:49:43.151345 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.151234    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c31f7bc6-8899-48f5-99bb-c8e7721528bd-kserve-provision-location\") pod \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\" (UID: \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\") "
Sep 08 13:49:43.151345 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.151287    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c31f7bc6-8899-48f5-99bb-c8e7721528bd-cabundle-cert\") pod \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\" (UID: \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\") "
Sep 08 13:49:43.151345 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.151338    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5r8qg\" (UniqueName: \"kubernetes.io/projected/c31f7bc6-8899-48f5-99bb-c8e7721528bd-kube-api-access-5r8qg\") pod \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\" (UID: \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\") "
Sep 08 13:49:43.151662 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.151385    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c31f7bc6-8899-48f5-99bb-c8e7721528bd-model-cache\") pod \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\" (UID: \"c31f7bc6-8899-48f5-99bb-c8e7721528bd\") "
Sep 08 13:49:43.151977 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.151906    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31f7bc6-8899-48f5-99bb-c8e7721528bd-home" (OuterVolumeSpecName: "home") pod "c31f7bc6-8899-48f5-99bb-c8e7721528bd" (UID: "c31f7bc6-8899-48f5-99bb-c8e7721528bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:43.152080 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.151973    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31f7bc6-8899-48f5-99bb-c8e7721528bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "c31f7bc6-8899-48f5-99bb-c8e7721528bd" (UID: "c31f7bc6-8899-48f5-99bb-c8e7721528bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:43.152839 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.152796    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c31f7bc6-8899-48f5-99bb-c8e7721528bd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c31f7bc6-8899-48f5-99bb-c8e7721528bd" (UID: "c31f7bc6-8899-48f5-99bb-c8e7721528bd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:49:43.153012 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.152985    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31f7bc6-8899-48f5-99bb-c8e7721528bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c31f7bc6-8899-48f5-99bb-c8e7721528bd" (UID: "c31f7bc6-8899-48f5-99bb-c8e7721528bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:43.155612 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.155542    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31f7bc6-8899-48f5-99bb-c8e7721528bd-dshm" (OuterVolumeSpecName: "dshm") pod "c31f7bc6-8899-48f5-99bb-c8e7721528bd" (UID: "c31f7bc6-8899-48f5-99bb-c8e7721528bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:43.155781 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.155750    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c31f7bc6-8899-48f5-99bb-c8e7721528bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c31f7bc6-8899-48f5-99bb-c8e7721528bd" (UID: "c31f7bc6-8899-48f5-99bb-c8e7721528bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:49:43.155844 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.155760    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c31f7bc6-8899-48f5-99bb-c8e7721528bd-kube-api-access-5r8qg" (OuterVolumeSpecName: "kube-api-access-5r8qg") pod "c31f7bc6-8899-48f5-99bb-c8e7721528bd" (UID: "c31f7bc6-8899-48f5-99bb-c8e7721528bd"). InnerVolumeSpecName "kube-api-access-5r8qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:49:43.175523 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.175487    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31f7bc6-8899-48f5-99bb-c8e7721528bd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c31f7bc6-8899-48f5-99bb-c8e7721528bd" (UID: "c31f7bc6-8899-48f5-99bb-c8e7721528bd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:43.227651 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.227586    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv"
Sep 08 13:49:43.252434 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.252412    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fd1f1bf-d2e7-4909-b815-12706acd0c13-kserve-provision-location\") pod \"1fd1f1bf-d2e7-4909-b815-12706acd0c13\" (UID: \"1fd1f1bf-d2e7-4909-b815-12706acd0c13\") "
Sep 08 13:49:43.252513 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.252446    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd1f1bf-d2e7-4909-b815-12706acd0c13-tls-certs\") pod \"1fd1f1bf-d2e7-4909-b815-12706acd0c13\" (UID: \"1fd1f1bf-d2e7-4909-b815-12706acd0c13\") "
Sep 08 13:49:43.252513 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.252476    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fd1f1bf-d2e7-4909-b815-12706acd0c13-cabundle-cert\") pod \"1fd1f1bf-d2e7-4909-b815-12706acd0c13\" (UID: \"1fd1f1bf-d2e7-4909-b815-12706acd0c13\") "
Sep 08 13:49:43.252614 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.252513    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fd1f1bf-d2e7-4909-b815-12706acd0c13-tmp-dir\") pod \"1fd1f1bf-d2e7-4909-b815-12706acd0c13\" (UID: \"1fd1f1bf-d2e7-4909-b815-12706acd0c13\") "
Sep 08 13:49:43.252797 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.252775    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c31f7bc6-8899-48f5-99bb-c8e7721528bd-cabundle-cert\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:49:43.252880 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.252802    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r8qg\" (UniqueName: \"kubernetes.io/projected/c31f7bc6-8899-48f5-99bb-c8e7721528bd-kube-api-access-5r8qg\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:49:43.252880 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.252816    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c31f7bc6-8899-48f5-99bb-c8e7721528bd-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:49:43.252880 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.252829    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c31f7bc6-8899-48f5-99bb-c8e7721528bd-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:49:43.252880 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.252840    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c31f7bc6-8899-48f5-99bb-c8e7721528bd-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:49:43.252880 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.252852    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c31f7bc6-8899-48f5-99bb-c8e7721528bd-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:49:43.252880 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.252865    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c31f7bc6-8899-48f5-99bb-c8e7721528bd-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:49:43.252880 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.252879    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c31f7bc6-8899-48f5-99bb-c8e7721528bd-kserve-provision-location\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:49:43.253118 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.252898    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fd1f1bf-d2e7-4909-b815-12706acd0c13-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1fd1f1bf-d2e7-4909-b815-12706acd0c13" (UID: "1fd1f1bf-d2e7-4909-b815-12706acd0c13"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:49:43.253118 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.252993    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fd1f1bf-d2e7-4909-b815-12706acd0c13-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1fd1f1bf-d2e7-4909-b815-12706acd0c13" (UID: "1fd1f1bf-d2e7-4909-b815-12706acd0c13"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:43.254474 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.254453    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fd1f1bf-d2e7-4909-b815-12706acd0c13-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1fd1f1bf-d2e7-4909-b815-12706acd0c13" (UID: "1fd1f1bf-d2e7-4909-b815-12706acd0c13"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:49:43.273739 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.273707    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fd1f1bf-d2e7-4909-b815-12706acd0c13-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1fd1f1bf-d2e7-4909-b815-12706acd0c13" (UID: "1fd1f1bf-d2e7-4909-b815-12706acd0c13"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:49:43.353513 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.353480    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fd1f1bf-d2e7-4909-b815-12706acd0c13-kserve-provision-location\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:49:43.353513 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.353510    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fd1f1bf-d2e7-4909-b815-12706acd0c13-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:49:43.353513 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.353521    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fd1f1bf-d2e7-4909-b815-12706acd0c13-cabundle-cert\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:49:43.353753 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.353529    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fd1f1bf-d2e7-4909-b815-12706acd0c13-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:49:43.773220 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.773186    2558 generic.go:356] "Generic (PLEG): container finished" podID="1fd1f1bf-d2e7-4909-b815-12706acd0c13" containerID="a7069bf0f1f33f9a96e6676f1634ea06eee1bbe94c39cc698f47c7058471401f" exitCode=0
Sep 08 13:49:43.773725 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.773240    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv"
Sep 08 13:49:43.773725 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.773277    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv" event={"ID":"1fd1f1bf-d2e7-4909-b815-12706acd0c13","Type":"ContainerDied","Data":"a7069bf0f1f33f9a96e6676f1634ea06eee1bbe94c39cc698f47c7058471401f"}
Sep 08 13:49:43.773725 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.773350    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv" event={"ID":"1fd1f1bf-d2e7-4909-b815-12706acd0c13","Type":"ContainerDied","Data":"e67be1be529d1bd7f3158d472440f7314df15f3f00027bb992a17c3a2f68f128"}
Sep 08 13:49:43.773725 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.773387    2558 scope.go:117] "RemoveContainer" containerID="a7069bf0f1f33f9a96e6676f1634ea06eee1bbe94c39cc698f47c7058471401f"
Sep 08 13:49:43.775125 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.775101    2558 generic.go:356] "Generic (PLEG): container finished" podID="c31f7bc6-8899-48f5-99bb-c8e7721528bd" containerID="58f3c4946501d1bf1eee79530c4f47e76f4d964e30d2731f5ec59bc93ce93398" exitCode=0
Sep 08 13:49:43.775228 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.775169    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"
Sep 08 13:49:43.775228 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.775198    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl" event={"ID":"c31f7bc6-8899-48f5-99bb-c8e7721528bd","Type":"ContainerDied","Data":"58f3c4946501d1bf1eee79530c4f47e76f4d964e30d2731f5ec59bc93ce93398"}
Sep 08 13:49:43.775307 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.775235    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl" event={"ID":"c31f7bc6-8899-48f5-99bb-c8e7721528bd","Type":"ContainerDied","Data":"ceeacb5012d0146d281b9096b8b57c8bd137cc34c6a5d84b7492fb1434c3b0cf"}
Sep 08 13:49:43.785859 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.785817    2558 scope.go:117] "RemoveContainer" containerID="5046ee26fc699bdb4867bc39d03df4d367750fa5152586d03898f694fdff6d1c"
Sep 08 13:49:43.794207 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.794190    2558 scope.go:117] "RemoveContainer" containerID="a7069bf0f1f33f9a96e6676f1634ea06eee1bbe94c39cc698f47c7058471401f"
Sep 08 13:49:43.794506 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:49:43.794489    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7069bf0f1f33f9a96e6676f1634ea06eee1bbe94c39cc698f47c7058471401f\": container with ID starting with a7069bf0f1f33f9a96e6676f1634ea06eee1bbe94c39cc698f47c7058471401f not found: ID does not exist" containerID="a7069bf0f1f33f9a96e6676f1634ea06eee1bbe94c39cc698f47c7058471401f"
Sep 08 13:49:43.794585 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.794516    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7069bf0f1f33f9a96e6676f1634ea06eee1bbe94c39cc698f47c7058471401f"} err="failed to get container status \"a7069bf0f1f33f9a96e6676f1634ea06eee1bbe94c39cc698f47c7058471401f\": rpc error: code = NotFound desc = could not find container \"a7069bf0f1f33f9a96e6676f1634ea06eee1bbe94c39cc698f47c7058471401f\": container with ID starting with a7069bf0f1f33f9a96e6676f1634ea06eee1bbe94c39cc698f47c7058471401f not found: ID does not exist"
Sep 08 13:49:43.794585 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.794539    2558 scope.go:117] "RemoveContainer" containerID="5046ee26fc699bdb4867bc39d03df4d367750fa5152586d03898f694fdff6d1c"
Sep 08 13:49:43.794903 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:49:43.794883    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5046ee26fc699bdb4867bc39d03df4d367750fa5152586d03898f694fdff6d1c\": container with ID starting with 5046ee26fc699bdb4867bc39d03df4d367750fa5152586d03898f694fdff6d1c not found: ID does not exist" containerID="5046ee26fc699bdb4867bc39d03df4d367750fa5152586d03898f694fdff6d1c"
Sep 08 13:49:43.794962 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.794911    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5046ee26fc699bdb4867bc39d03df4d367750fa5152586d03898f694fdff6d1c"} err="failed to get container status \"5046ee26fc699bdb4867bc39d03df4d367750fa5152586d03898f694fdff6d1c\": rpc error: code = NotFound desc = could not find container \"5046ee26fc699bdb4867bc39d03df4d367750fa5152586d03898f694fdff6d1c\": container with ID starting with 5046ee26fc699bdb4867bc39d03df4d367750fa5152586d03898f694fdff6d1c not found: ID does not exist"
Sep 08 13:49:43.794962 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.794928    2558 scope.go:117] "RemoveContainer" containerID="58f3c4946501d1bf1eee79530c4f47e76f4d964e30d2731f5ec59bc93ce93398"
Sep 08 13:49:43.802976 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.802961    2558 scope.go:117] "RemoveContainer" containerID="a940a597561af89f2cfff3e27781701bcdac17572150b4802fbcb1ddba68aff0"
Sep 08 13:49:43.810669 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.810577    2558 scope.go:117] "RemoveContainer" containerID="58f3c4946501d1bf1eee79530c4f47e76f4d964e30d2731f5ec59bc93ce93398"
Sep 08 13:49:43.810955 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:49:43.810922    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58f3c4946501d1bf1eee79530c4f47e76f4d964e30d2731f5ec59bc93ce93398\": container with ID starting with 58f3c4946501d1bf1eee79530c4f47e76f4d964e30d2731f5ec59bc93ce93398 not found: ID does not exist" containerID="58f3c4946501d1bf1eee79530c4f47e76f4d964e30d2731f5ec59bc93ce93398"
Sep 08 13:49:43.811055 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.810966    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58f3c4946501d1bf1eee79530c4f47e76f4d964e30d2731f5ec59bc93ce93398"} err="failed to get container status \"58f3c4946501d1bf1eee79530c4f47e76f4d964e30d2731f5ec59bc93ce93398\": rpc error: code = NotFound desc = could not find container \"58f3c4946501d1bf1eee79530c4f47e76f4d964e30d2731f5ec59bc93ce93398\": container with ID starting with 58f3c4946501d1bf1eee79530c4f47e76f4d964e30d2731f5ec59bc93ce93398 not found: ID does not exist"
Sep 08 13:49:43.811055 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.810988    2558 scope.go:117] "RemoveContainer" containerID="a940a597561af89f2cfff3e27781701bcdac17572150b4802fbcb1ddba68aff0"
Sep 08 13:49:43.811334 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:49:43.811315    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a940a597561af89f2cfff3e27781701bcdac17572150b4802fbcb1ddba68aff0\": container with ID starting with a940a597561af89f2cfff3e27781701bcdac17572150b4802fbcb1ddba68aff0 not found: ID does not exist" containerID="a940a597561af89f2cfff3e27781701bcdac17572150b4802fbcb1ddba68aff0"
Sep 08 13:49:43.811449 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.811342    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a940a597561af89f2cfff3e27781701bcdac17572150b4802fbcb1ddba68aff0"} err="failed to get container status \"a940a597561af89f2cfff3e27781701bcdac17572150b4802fbcb1ddba68aff0\": rpc error: code = NotFound desc = could not find container \"a940a597561af89f2cfff3e27781701bcdac17572150b4802fbcb1ddba68aff0\": container with ID starting with a940a597561af89f2cfff3e27781701bcdac17572150b4802fbcb1ddba68aff0 not found: ID does not exist"
Sep 08 13:49:43.813070 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.813050    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"]
Sep 08 13:49:43.820323 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.820303    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-559469c4b5-mrqvl"]
Sep 08 13:49:43.837317 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.837295    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv"]
Sep 08 13:49:43.840413 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:43.840390    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-6b5fc995f6-94fdv"]
Sep 08 13:49:45.053315 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:45.053279    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fd1f1bf-d2e7-4909-b815-12706acd0c13" path="/var/lib/kubelet/pods/1fd1f1bf-d2e7-4909-b815-12706acd0c13/volumes"
Sep 08 13:49:45.053779 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:49:45.053765    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c31f7bc6-8899-48f5-99bb-c8e7721528bd" path="/var/lib/kubelet/pods/c31f7bc6-8899-48f5-99bb-c8e7721528bd/volumes"
Sep 08 13:50:15.363678 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.363646    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb"]
Sep 08 13:50:15.364105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.364006    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c31f7bc6-8899-48f5-99bb-c8e7721528bd" containerName="storage-initializer"
Sep 08 13:50:15.364105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.364018    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c31f7bc6-8899-48f5-99bb-c8e7721528bd" containerName="storage-initializer"
Sep 08 13:50:15.364105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.364032    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c31f7bc6-8899-48f5-99bb-c8e7721528bd" containerName="main"
Sep 08 13:50:15.364105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.364038    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c31f7bc6-8899-48f5-99bb-c8e7721528bd" containerName="main"
Sep 08 13:50:15.364105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.364049    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eaf0b84-1b6f-4976-b566-ecc1115e6775" containerName="main"
Sep 08 13:50:15.364105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.364055    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eaf0b84-1b6f-4976-b566-ecc1115e6775" containerName="main"
Sep 08 13:50:15.364105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.364062    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fd1f1bf-d2e7-4909-b815-12706acd0c13" containerName="storage-initializer"
Sep 08 13:50:15.364105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.364068    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fd1f1bf-d2e7-4909-b815-12706acd0c13" containerName="storage-initializer"
Sep 08 13:50:15.364105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.364077    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fd1f1bf-d2e7-4909-b815-12706acd0c13" containerName="main"
Sep 08 13:50:15.364105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.364083    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fd1f1bf-d2e7-4909-b815-12706acd0c13" containerName="main"
Sep 08 13:50:15.364105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.364098    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23a480c3-479b-4189-a996-a5a9b0651139" containerName="main"
Sep 08 13:50:15.364105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.364103    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="23a480c3-479b-4189-a996-a5a9b0651139" containerName="main"
Sep 08 13:50:15.364480 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.364153    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="23a480c3-479b-4189-a996-a5a9b0651139" containerName="main"
Sep 08 13:50:15.364480 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.364165    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eaf0b84-1b6f-4976-b566-ecc1115e6775" containerName="main"
Sep 08 13:50:15.364480 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.364170    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c31f7bc6-8899-48f5-99bb-c8e7721528bd" containerName="main"
Sep 08 13:50:15.364480 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.364177    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fd1f1bf-d2e7-4909-b815-12706acd0c13" containerName="main"
Sep 08 13:50:15.367246 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.367229    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb"
Sep 08 13:50:15.371932 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.371905    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"openshift-service-ca.crt\""
Sep 08 13:50:15.373128 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.373098    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v1-kserve-self-signed-certs\""
Sep 08 13:50:15.373303 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.373108    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"kube-root-ca.crt\""
Sep 08 13:50:15.373303 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.373108    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"default-dockercfg-9gr84\""
Sep 08 13:50:15.382703 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.382676    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb"]
Sep 08 13:50:15.415257 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.415222    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d58c506e-6fd1-4e67-9f61-ba89882c787f-model-cache\") pod \"decom-v1-kserve-6cb797fb75-5ptdb\" (UID: \"d58c506e-6fd1-4e67-9f61-ba89882c787f\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb"
Sep 08 13:50:15.415257 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.415261    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d58c506e-6fd1-4e67-9f61-ba89882c787f-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-5ptdb\" (UID: \"d58c506e-6fd1-4e67-9f61-ba89882c787f\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb"
Sep 08 13:50:15.415460 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.415276    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d58c506e-6fd1-4e67-9f61-ba89882c787f-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-5ptdb\" (UID: \"d58c506e-6fd1-4e67-9f61-ba89882c787f\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb"
Sep 08 13:50:15.415460 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.415295    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d58c506e-6fd1-4e67-9f61-ba89882c787f-home\") pod \"decom-v1-kserve-6cb797fb75-5ptdb\" (UID: \"d58c506e-6fd1-4e67-9f61-ba89882c787f\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb"
Sep 08 13:50:15.415460 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.415353    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d58c506e-6fd1-4e67-9f61-ba89882c787f-dshm\") pod \"decom-v1-kserve-6cb797fb75-5ptdb\" (UID: \"d58c506e-6fd1-4e67-9f61-ba89882c787f\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb"
Sep 08 13:50:15.415460 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.415389    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf788\" (UniqueName: \"kubernetes.io/projected/d58c506e-6fd1-4e67-9f61-ba89882c787f-kube-api-access-wf788\") pod \"decom-v1-kserve-6cb797fb75-5ptdb\" (UID: \"d58c506e-6fd1-4e67-9f61-ba89882c787f\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb"
Sep 08 13:50:15.516422 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.516366    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d58c506e-6fd1-4e67-9f61-ba89882c787f-dshm\") pod \"decom-v1-kserve-6cb797fb75-5ptdb\" (UID: \"d58c506e-6fd1-4e67-9f61-ba89882c787f\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb"
Sep 08 13:50:15.516643 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.516441    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf788\" (UniqueName: \"kubernetes.io/projected/d58c506e-6fd1-4e67-9f61-ba89882c787f-kube-api-access-wf788\") pod \"decom-v1-kserve-6cb797fb75-5ptdb\" (UID: \"d58c506e-6fd1-4e67-9f61-ba89882c787f\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb"
Sep 08 13:50:15.516643 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.516528    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d58c506e-6fd1-4e67-9f61-ba89882c787f-model-cache\") pod \"decom-v1-kserve-6cb797fb75-5ptdb\" (UID: \"d58c506e-6fd1-4e67-9f61-ba89882c787f\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb"
Sep 08 13:50:15.516643 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.516562    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d58c506e-6fd1-4e67-9f61-ba89882c787f-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-5ptdb\" (UID: \"d58c506e-6fd1-4e67-9f61-ba89882c787f\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb"
Sep 08 13:50:15.516643 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.516581    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d58c506e-6fd1-4e67-9f61-ba89882c787f-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-5ptdb\" (UID: \"d58c506e-6fd1-4e67-9f61-ba89882c787f\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb"
Sep 08 13:50:15.516643 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.516634    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d58c506e-6fd1-4e67-9f61-ba89882c787f-home\") pod \"decom-v1-kserve-6cb797fb75-5ptdb\" (UID: \"d58c506e-6fd1-4e67-9f61-ba89882c787f\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb"
Sep 08 13:50:15.516994 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.516970    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d58c506e-6fd1-4e67-9f61-ba89882c787f-model-cache\") pod \"decom-v1-kserve-6cb797fb75-5ptdb\" (UID: \"d58c506e-6fd1-4e67-9f61-ba89882c787f\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb"
Sep 08 13:50:15.516994 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.516986    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d58c506e-6fd1-4e67-9f61-ba89882c787f-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-5ptdb\" (UID: \"d58c506e-6fd1-4e67-9f61-ba89882c787f\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb"
Sep 08 13:50:15.517117 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.517013    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d58c506e-6fd1-4e67-9f61-ba89882c787f-home\") pod \"decom-v1-kserve-6cb797fb75-5ptdb\" (UID: \"d58c506e-6fd1-4e67-9f61-ba89882c787f\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb"
Sep 08 13:50:15.518760 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.518738    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d58c506e-6fd1-4e67-9f61-ba89882c787f-dshm\") pod \"decom-v1-kserve-6cb797fb75-5ptdb\" (UID: \"d58c506e-6fd1-4e67-9f61-ba89882c787f\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb"
Sep 08 13:50:15.519071 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.519054    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d58c506e-6fd1-4e67-9f61-ba89882c787f-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-5ptdb\" (UID: \"d58c506e-6fd1-4e67-9f61-ba89882c787f\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb"
Sep 08 13:50:15.529293 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.529268    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf788\" (UniqueName: \"kubernetes.io/projected/d58c506e-6fd1-4e67-9f61-ba89882c787f-kube-api-access-wf788\") pod \"decom-v1-kserve-6cb797fb75-5ptdb\" (UID: \"d58c506e-6fd1-4e67-9f61-ba89882c787f\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb"
Sep 08 13:50:15.678392 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.678294    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb"
Sep 08 13:50:15.961379 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:15.961331    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb"]
Sep 08 13:50:15.964362 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:50:15.964331    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd58c506e_6fd1_4e67_9f61_ba89882c787f.slice/crio-83605c5ced718951e7c7a6b8714a5afb524609c169a52f9a8718c658351987e8 WatchSource:0}: Error finding container 83605c5ced718951e7c7a6b8714a5afb524609c169a52f9a8718c658351987e8: Status 404 returned error can't find the container with id 83605c5ced718951e7c7a6b8714a5afb524609c169a52f9a8718c658351987e8
Sep 08 13:50:16.941937 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:16.941895    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb" event={"ID":"d58c506e-6fd1-4e67-9f61-ba89882c787f","Type":"ContainerStarted","Data":"486f8dc06c3a688b863ba49a998eaf83fe9122652d00661a24146511c2dea0ee"}
Sep 08 13:50:16.942343 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:16.941948    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb" event={"ID":"d58c506e-6fd1-4e67-9f61-ba89882c787f","Type":"ContainerStarted","Data":"83605c5ced718951e7c7a6b8714a5afb524609c169a52f9a8718c658351987e8"}
Sep 08 13:50:16.971983 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:16.971913    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb" podStartSLOduration=1.9718919019999999 podStartE2EDuration="1.971891902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:50:16.971753827 +0000 UTC m=+2080.423514757" watchObservedRunningTime="2026-09-08 13:50:16.971891902 +0000 UTC m=+2080.423652838"
Sep 08 13:50:17.678434 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:17.678396    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb"
Sep 08 13:50:21.318317 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:21.318279    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq"]
Sep 08 13:50:21.321959 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:21.321936    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq"
Sep 08 13:50:21.324720 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:21.324696    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v2-kserve-self-signed-certs\""
Sep 08 13:50:21.338864 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:21.338839    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq"]
Sep 08 13:50:21.476062 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:21.476024    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfsz2\" (UniqueName: \"kubernetes.io/projected/091c6742-9a7f-4bd0-8df1-1c49b161d79c-kube-api-access-mfsz2\") pod \"decom-v2-kserve-77f867d787-cq7bq\" (UID: \"091c6742-9a7f-4bd0-8df1-1c49b161d79c\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq"
Sep 08 13:50:21.476062 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:21.476066    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/091c6742-9a7f-4bd0-8df1-1c49b161d79c-model-cache\") pod \"decom-v2-kserve-77f867d787-cq7bq\" (UID: \"091c6742-9a7f-4bd0-8df1-1c49b161d79c\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq"
Sep 08 13:50:21.476345 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:21.476090    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/091c6742-9a7f-4bd0-8df1-1c49b161d79c-home\") pod \"decom-v2-kserve-77f867d787-cq7bq\" (UID: \"091c6742-9a7f-4bd0-8df1-1c49b161d79c\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq"
Sep 08 13:50:21.476345 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:21.476219    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/091c6742-9a7f-4bd0-8df1-1c49b161d79c-tls-certs\") pod \"decom-v2-kserve-77f867d787-cq7bq\" (UID: \"091c6742-9a7f-4bd0-8df1-1c49b161d79c\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq"
Sep 08 13:50:21.476345 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:21.476276    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/091c6742-9a7f-4bd0-8df1-1c49b161d79c-dshm\") pod \"decom-v2-kserve-77f867d787-cq7bq\" (UID: \"091c6742-9a7f-4bd0-8df1-1c49b161d79c\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq"
Sep 08 13:50:21.476345 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:21.476316    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/091c6742-9a7f-4bd0-8df1-1c49b161d79c-tmp-dir\") pod \"decom-v2-kserve-77f867d787-cq7bq\" (UID: \"091c6742-9a7f-4bd0-8df1-1c49b161d79c\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq"
Sep 08 13:50:21.577108 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:21.576992    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/091c6742-9a7f-4bd0-8df1-1c49b161d79c-tmp-dir\") pod \"decom-v2-kserve-77f867d787-cq7bq\" (UID: \"091c6742-9a7f-4bd0-8df1-1c49b161d79c\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq"
Sep 08 13:50:21.577108 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:21.577095    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfsz2\" (UniqueName: \"kubernetes.io/projected/091c6742-9a7f-4bd0-8df1-1c49b161d79c-kube-api-access-mfsz2\") pod \"decom-v2-kserve-77f867d787-cq7bq\" (UID: \"091c6742-9a7f-4bd0-8df1-1c49b161d79c\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq"
Sep 08 13:50:21.577349 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:21.577132    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/091c6742-9a7f-4bd0-8df1-1c49b161d79c-model-cache\") pod \"decom-v2-kserve-77f867d787-cq7bq\" (UID: \"091c6742-9a7f-4bd0-8df1-1c49b161d79c\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq"
Sep 08 13:50:21.577349 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:21.577164    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/091c6742-9a7f-4bd0-8df1-1c49b161d79c-home\") pod \"decom-v2-kserve-77f867d787-cq7bq\" (UID: \"091c6742-9a7f-4bd0-8df1-1c49b161d79c\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq"
Sep 08 13:50:21.577349 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:21.577234    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/091c6742-9a7f-4bd0-8df1-1c49b161d79c-tls-certs\") pod \"decom-v2-kserve-77f867d787-cq7bq\" (UID: \"091c6742-9a7f-4bd0-8df1-1c49b161d79c\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq"
Sep 08 13:50:21.577349 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:21.577284    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/091c6742-9a7f-4bd0-8df1-1c49b161d79c-dshm\") pod \"decom-v2-kserve-77f867d787-cq7bq\" (UID: \"091c6742-9a7f-4bd0-8df1-1c49b161d79c\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq"
Sep 08 13:50:21.577570 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:21.577514    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/091c6742-9a7f-4bd0-8df1-1c49b161d79c-tmp-dir\") pod \"decom-v2-kserve-77f867d787-cq7bq\" (UID: \"091c6742-9a7f-4bd0-8df1-1c49b161d79c\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq"
Sep 08 13:50:21.577864 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:21.577835    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/091c6742-9a7f-4bd0-8df1-1c49b161d79c-model-cache\") pod \"decom-v2-kserve-77f867d787-cq7bq\" (UID: \"091c6742-9a7f-4bd0-8df1-1c49b161d79c\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq"
Sep 08 13:50:21.578021 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:21.577946    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/091c6742-9a7f-4bd0-8df1-1c49b161d79c-home\") pod \"decom-v2-kserve-77f867d787-cq7bq\" (UID: \"091c6742-9a7f-4bd0-8df1-1c49b161d79c\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq"
Sep 08 13:50:21.580230 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:21.580192    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/091c6742-9a7f-4bd0-8df1-1c49b161d79c-dshm\") pod \"decom-v2-kserve-77f867d787-cq7bq\" (UID: \"091c6742-9a7f-4bd0-8df1-1c49b161d79c\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq"
Sep 08 13:50:21.580953 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:21.580930    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/091c6742-9a7f-4bd0-8df1-1c49b161d79c-tls-certs\") pod \"decom-v2-kserve-77f867d787-cq7bq\" (UID: \"091c6742-9a7f-4bd0-8df1-1c49b161d79c\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq"
Sep 08 13:50:21.588556 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:21.588529    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfsz2\" (UniqueName: \"kubernetes.io/projected/091c6742-9a7f-4bd0-8df1-1c49b161d79c-kube-api-access-mfsz2\") pod \"decom-v2-kserve-77f867d787-cq7bq\" (UID: \"091c6742-9a7f-4bd0-8df1-1c49b161d79c\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq"
Sep 08 13:50:21.633143 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:21.633099    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq"
Sep 08 13:50:21.914038 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:21.914000    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq"]
Sep 08 13:50:21.918708 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:50:21.918666    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod091c6742_9a7f_4bd0_8df1_1c49b161d79c.slice/crio-6e972416d93f07d85cb00e36381a6c7c33b9620bd95b9d44088a32108c754f3b WatchSource:0}: Error finding container 6e972416d93f07d85cb00e36381a6c7c33b9620bd95b9d44088a32108c754f3b: Status 404 returned error can't find the container with id 6e972416d93f07d85cb00e36381a6c7c33b9620bd95b9d44088a32108c754f3b
Sep 08 13:50:21.970222 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:21.970181    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq" event={"ID":"091c6742-9a7f-4bd0-8df1-1c49b161d79c","Type":"ContainerStarted","Data":"6e972416d93f07d85cb00e36381a6c7c33b9620bd95b9d44088a32108c754f3b"}
Sep 08 13:50:22.976464 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:22.976428    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq" event={"ID":"091c6742-9a7f-4bd0-8df1-1c49b161d79c","Type":"ContainerStarted","Data":"50fd58e43fdd961d0f61bc70f3758aeea0dcb021cc68c43682b80305bd6fdeff"}
Sep 08 13:50:23.003449 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:23.003387    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq" podStartSLOduration=2.003370912 podStartE2EDuration="2.003370912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:50:23.00177129 +0000 UTC m=+2086.453532231" watchObservedRunningTime="2026-09-08 13:50:23.003370912 +0000 UTC m=+2086.455131843"
Sep 08 13:50:23.633925 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:23.633871    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq"
Sep 08 13:50:25.679041 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:25.678996    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb"
Sep 08 13:50:25.692023 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:25.691990    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb"
Sep 08 13:50:26.001815 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:26.001727    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb"
Sep 08 13:50:31.634080 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:31.634041    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq"
Sep 08 13:50:31.646633 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:31.646581    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq"
Sep 08 13:50:32.030260 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:32.030169    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq"
Sep 08 13:50:50.550321 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.550287    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"]
Sep 08 13:50:50.554026 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.554010    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"
Sep 08 13:50:50.556936 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.556904    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 08 13:50:50.557089 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.557069    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:50:50.557938 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.557918    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-6r98f\""
Sep 08 13:50:50.558049 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.557956    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 08 13:50:50.559317 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.559298    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 08 13:50:50.559317 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.559321    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 08 13:50:50.570454 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.570433    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"]
Sep 08 13:50:50.627487 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.627452    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5560342-e146-4c6b-b5df-f94a34b13276-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2\" (UID: \"f5560342-e146-4c6b-b5df-f94a34b13276\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"
Sep 08 13:50:50.627659 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.627498    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5560342-e146-4c6b-b5df-f94a34b13276-tls-certs\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2\" (UID: \"f5560342-e146-4c6b-b5df-f94a34b13276\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"
Sep 08 13:50:50.627659 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.627557    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9htq\" (UniqueName: \"kubernetes.io/projected/f5560342-e146-4c6b-b5df-f94a34b13276-kube-api-access-q9htq\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2\" (UID: \"f5560342-e146-4c6b-b5df-f94a34b13276\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"
Sep 08 13:50:50.627659 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.627617    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5560342-e146-4c6b-b5df-f94a34b13276-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2\" (UID: \"f5560342-e146-4c6b-b5df-f94a34b13276\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"
Sep 08 13:50:50.627659 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.627636    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5560342-e146-4c6b-b5df-f94a34b13276-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2\" (UID: \"f5560342-e146-4c6b-b5df-f94a34b13276\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"
Sep 08 13:50:50.627807 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.627669    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5560342-e146-4c6b-b5df-f94a34b13276-model-cache\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2\" (UID: \"f5560342-e146-4c6b-b5df-f94a34b13276\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"
Sep 08 13:50:50.627807 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.627708    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5560342-e146-4c6b-b5df-f94a34b13276-home\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2\" (UID: \"f5560342-e146-4c6b-b5df-f94a34b13276\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"
Sep 08 13:50:50.627807 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.627732    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5560342-e146-4c6b-b5df-f94a34b13276-dshm\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2\" (UID: \"f5560342-e146-4c6b-b5df-f94a34b13276\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"
Sep 08 13:50:50.729159 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.729124    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5560342-e146-4c6b-b5df-f94a34b13276-dshm\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2\" (UID: \"f5560342-e146-4c6b-b5df-f94a34b13276\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"
Sep 08 13:50:50.729345 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.729203    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5560342-e146-4c6b-b5df-f94a34b13276-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2\" (UID: \"f5560342-e146-4c6b-b5df-f94a34b13276\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"
Sep 08 13:50:50.729345 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.729226    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5560342-e146-4c6b-b5df-f94a34b13276-tls-certs\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2\" (UID: \"f5560342-e146-4c6b-b5df-f94a34b13276\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"
Sep 08 13:50:50.729345 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.729248    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9htq\" (UniqueName: \"kubernetes.io/projected/f5560342-e146-4c6b-b5df-f94a34b13276-kube-api-access-q9htq\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2\" (UID: \"f5560342-e146-4c6b-b5df-f94a34b13276\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"
Sep 08 13:50:50.729345 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.729269    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5560342-e146-4c6b-b5df-f94a34b13276-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2\" (UID: \"f5560342-e146-4c6b-b5df-f94a34b13276\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"
Sep 08 13:50:50.729345 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.729287    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5560342-e146-4c6b-b5df-f94a34b13276-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2\" (UID: \"f5560342-e146-4c6b-b5df-f94a34b13276\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"
Sep 08 13:50:50.729345 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.729314    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5560342-e146-4c6b-b5df-f94a34b13276-model-cache\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2\" (UID: \"f5560342-e146-4c6b-b5df-f94a34b13276\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"
Sep 08 13:50:50.729345 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.729346    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5560342-e146-4c6b-b5df-f94a34b13276-home\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2\" (UID: \"f5560342-e146-4c6b-b5df-f94a34b13276\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"
Sep 08 13:50:50.729788 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.729691    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5560342-e146-4c6b-b5df-f94a34b13276-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2\" (UID: \"f5560342-e146-4c6b-b5df-f94a34b13276\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"
Sep 08 13:50:50.729850 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.729813    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5560342-e146-4c6b-b5df-f94a34b13276-model-cache\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2\" (UID: \"f5560342-e146-4c6b-b5df-f94a34b13276\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"
Sep 08 13:50:50.729908 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.729895    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5560342-e146-4c6b-b5df-f94a34b13276-home\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2\" (UID: \"f5560342-e146-4c6b-b5df-f94a34b13276\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"
Sep 08 13:50:50.729951 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.729894    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5560342-e146-4c6b-b5df-f94a34b13276-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2\" (UID: \"f5560342-e146-4c6b-b5df-f94a34b13276\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"
Sep 08 13:50:50.730110 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.730086    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5560342-e146-4c6b-b5df-f94a34b13276-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2\" (UID: \"f5560342-e146-4c6b-b5df-f94a34b13276\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"
Sep 08 13:50:50.731503 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.731484    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5560342-e146-4c6b-b5df-f94a34b13276-dshm\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2\" (UID: \"f5560342-e146-4c6b-b5df-f94a34b13276\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"
Sep 08 13:50:50.732138 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.732116    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5560342-e146-4c6b-b5df-f94a34b13276-tls-certs\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2\" (UID: \"f5560342-e146-4c6b-b5df-f94a34b13276\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"
Sep 08 13:50:50.740574 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.740553    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9htq\" (UniqueName: \"kubernetes.io/projected/f5560342-e146-4c6b-b5df-f94a34b13276-kube-api-access-q9htq\") pod \"tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2\" (UID: \"f5560342-e146-4c6b-b5df-f94a34b13276\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"
Sep 08 13:50:50.807615 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.807532    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc"]
Sep 08 13:50:50.812235 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.812218    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc"
Sep 08 13:50:50.824795 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.824766    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc"]
Sep 08 13:50:50.830239 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.830213    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ac581a1-50f3-474b-995d-57c1813ae2a2-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc\" (UID: \"6ac581a1-50f3-474b-995d-57c1813ae2a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc"
Sep 08 13:50:50.830367 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.830262    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ac581a1-50f3-474b-995d-57c1813ae2a2-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc\" (UID: \"6ac581a1-50f3-474b-995d-57c1813ae2a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc"
Sep 08 13:50:50.830367 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.830339    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ac581a1-50f3-474b-995d-57c1813ae2a2-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc\" (UID: \"6ac581a1-50f3-474b-995d-57c1813ae2a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc"
Sep 08 13:50:50.830460 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.830444    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ac581a1-50f3-474b-995d-57c1813ae2a2-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc\" (UID: \"6ac581a1-50f3-474b-995d-57c1813ae2a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc"
Sep 08 13:50:50.864564 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.864532    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"
Sep 08 13:50:50.931384 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.931342    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ac581a1-50f3-474b-995d-57c1813ae2a2-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc\" (UID: \"6ac581a1-50f3-474b-995d-57c1813ae2a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc"
Sep 08 13:50:50.931557 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.931401    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ac581a1-50f3-474b-995d-57c1813ae2a2-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc\" (UID: \"6ac581a1-50f3-474b-995d-57c1813ae2a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc"
Sep 08 13:50:50.931557 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.931432    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ac581a1-50f3-474b-995d-57c1813ae2a2-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc\" (UID: \"6ac581a1-50f3-474b-995d-57c1813ae2a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc"
Sep 08 13:50:50.931557 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.931474    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ac581a1-50f3-474b-995d-57c1813ae2a2-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc\" (UID: \"6ac581a1-50f3-474b-995d-57c1813ae2a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc"
Sep 08 13:50:50.931948 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.931919    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ac581a1-50f3-474b-995d-57c1813ae2a2-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc\" (UID: \"6ac581a1-50f3-474b-995d-57c1813ae2a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc"
Sep 08 13:50:50.932118 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.932100    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ac581a1-50f3-474b-995d-57c1813ae2a2-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc\" (UID: \"6ac581a1-50f3-474b-995d-57c1813ae2a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc"
Sep 08 13:50:50.932327 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.932307    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ac581a1-50f3-474b-995d-57c1813ae2a2-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc\" (UID: \"6ac581a1-50f3-474b-995d-57c1813ae2a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc"
Sep 08 13:50:50.933958 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:50.933938    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ac581a1-50f3-474b-995d-57c1813ae2a2-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc\" (UID: \"6ac581a1-50f3-474b-995d-57c1813ae2a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc"
Sep 08 13:50:51.124076 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:51.124048    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc"
Sep 08 13:50:51.144233 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:51.144200    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"]
Sep 08 13:50:51.145695 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:50:51.145669    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5560342_e146_4c6b_b5df_f94a34b13276.slice/crio-9b59fc20d66db85c1533bd0540460130b8a320b298a04583a287a1d12c96a9fc WatchSource:0}: Error finding container 9b59fc20d66db85c1533bd0540460130b8a320b298a04583a287a1d12c96a9fc: Status 404 returned error can't find the container with id 9b59fc20d66db85c1533bd0540460130b8a320b298a04583a287a1d12c96a9fc
Sep 08 13:50:51.402406 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:51.402370    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc"]
Sep 08 13:50:51.403899 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:50:51.403868    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac581a1_50f3_474b_995d_57c1813ae2a2.slice/crio-d542e9bebd23da3b31bde70b7e00d9e18ec6f717a212e082e3890ed9718d5906 WatchSource:0}: Error finding container d542e9bebd23da3b31bde70b7e00d9e18ec6f717a212e082e3890ed9718d5906: Status 404 returned error can't find the container with id d542e9bebd23da3b31bde70b7e00d9e18ec6f717a212e082e3890ed9718d5906
Sep 08 13:50:52.123615 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:52.123559    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2" event={"ID":"f5560342-e146-4c6b-b5df-f94a34b13276","Type":"ContainerStarted","Data":"9ec66f0227d72da98c2ea8b026b623e975f940066409a2b6dc247b9e1253a249"}
Sep 08 13:50:52.124125 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:52.123662    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2" event={"ID":"f5560342-e146-4c6b-b5df-f94a34b13276","Type":"ContainerStarted","Data":"9b59fc20d66db85c1533bd0540460130b8a320b298a04583a287a1d12c96a9fc"}
Sep 08 13:50:52.126505 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:52.126477    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc" event={"ID":"6ac581a1-50f3-474b-995d-57c1813ae2a2","Type":"ContainerStarted","Data":"bf3881d3e62c3be6246cf03e200286b97f73e32d1cc0831fa702a0a7adc258c5"}
Sep 08 13:50:52.126690 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:52.126524    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc" event={"ID":"6ac581a1-50f3-474b-995d-57c1813ae2a2","Type":"ContainerStarted","Data":"d542e9bebd23da3b31bde70b7e00d9e18ec6f717a212e082e3890ed9718d5906"}
Sep 08 13:50:54.144604 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:54.144569    2558 generic.go:356] "Generic (PLEG): container finished" podID="f5560342-e146-4c6b-b5df-f94a34b13276" containerID="9ec66f0227d72da98c2ea8b026b623e975f940066409a2b6dc247b9e1253a249" exitCode=0
Sep 08 13:50:54.145094 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:54.144652    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2" event={"ID":"f5560342-e146-4c6b-b5df-f94a34b13276","Type":"ContainerDied","Data":"9ec66f0227d72da98c2ea8b026b623e975f940066409a2b6dc247b9e1253a249"}
Sep 08 13:50:54.146330 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:54.146309    2558 generic.go:356] "Generic (PLEG): container finished" podID="6ac581a1-50f3-474b-995d-57c1813ae2a2" containerID="bf3881d3e62c3be6246cf03e200286b97f73e32d1cc0831fa702a0a7adc258c5" exitCode=0
Sep 08 13:50:54.146445 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:54.146385    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc" event={"ID":"6ac581a1-50f3-474b-995d-57c1813ae2a2","Type":"ContainerDied","Data":"bf3881d3e62c3be6246cf03e200286b97f73e32d1cc0831fa702a0a7adc258c5"}
Sep 08 13:50:54.446701 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:54.446614    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq"]
Sep 08 13:50:55.155549 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:55.155511    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2" event={"ID":"f5560342-e146-4c6b-b5df-f94a34b13276","Type":"ContainerStarted","Data":"42bdb65a0b3f509d07a48accc6818b92fe077710274a25ce5aa5b3e5156a77e6"}
Sep 08 13:50:55.157518 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:55.157494    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc" event={"ID":"6ac581a1-50f3-474b-995d-57c1813ae2a2","Type":"ContainerStarted","Data":"353966e282cc95ff952294cba6da39af254a3993a264cfae909473f45f9cba49"}
Sep 08 13:50:55.157733 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:55.157717    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc"
Sep 08 13:50:55.183517 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:55.183474    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2" podStartSLOduration=2.579131804 podStartE2EDuration="5.183461742s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.604329938s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:50:55.182100115 +0000 UTC m=+2118.633861056" watchObservedRunningTime="2026-09-08 13:50:55.183461742 +0000 UTC m=+2118.635222672"
Sep 08 13:50:55.203526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:55.203478    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc" podStartSLOduration=2.628180607 podStartE2EDuration="5.203465234s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.575284627s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:50:55.202794747 +0000 UTC m=+2118.654555679" watchObservedRunningTime="2026-09-08 13:50:55.203465234 +0000 UTC m=+2118.655226166"
Sep 08 13:50:55.865086 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:50:55.865051    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"
Sep 08 13:51:00.865148 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:00.865090    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"
Sep 08 13:51:00.877783 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:00.877757    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"
Sep 08 13:51:01.200381 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:01.200298    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"
Sep 08 13:51:09.466550 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:09.466476    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq" podUID="091c6742-9a7f-4bd0-8df1-1c49b161d79c" containerName="main" containerID="cri-o://50fd58e43fdd961d0f61bc70f3758aeea0dcb021cc68c43682b80305bd6fdeff" gracePeriod=45
Sep 08 13:51:09.726551 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:09.726489    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq"
Sep 08 13:51:09.812479 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:09.812443    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/091c6742-9a7f-4bd0-8df1-1c49b161d79c-model-cache\") pod \"091c6742-9a7f-4bd0-8df1-1c49b161d79c\" (UID: \"091c6742-9a7f-4bd0-8df1-1c49b161d79c\") "
Sep 08 13:51:09.812665 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:09.812489    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/091c6742-9a7f-4bd0-8df1-1c49b161d79c-tls-certs\") pod \"091c6742-9a7f-4bd0-8df1-1c49b161d79c\" (UID: \"091c6742-9a7f-4bd0-8df1-1c49b161d79c\") "
Sep 08 13:51:09.812665 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:09.812550    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/091c6742-9a7f-4bd0-8df1-1c49b161d79c-home\") pod \"091c6742-9a7f-4bd0-8df1-1c49b161d79c\" (UID: \"091c6742-9a7f-4bd0-8df1-1c49b161d79c\") "
Sep 08 13:51:09.812665 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:09.812574    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/091c6742-9a7f-4bd0-8df1-1c49b161d79c-dshm\") pod \"091c6742-9a7f-4bd0-8df1-1c49b161d79c\" (UID: \"091c6742-9a7f-4bd0-8df1-1c49b161d79c\") "
Sep 08 13:51:09.812665 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:09.812627    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfsz2\" (UniqueName: \"kubernetes.io/projected/091c6742-9a7f-4bd0-8df1-1c49b161d79c-kube-api-access-mfsz2\") pod \"091c6742-9a7f-4bd0-8df1-1c49b161d79c\" (UID: \"091c6742-9a7f-4bd0-8df1-1c49b161d79c\") "
Sep 08 13:51:09.812665 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:09.812659    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/091c6742-9a7f-4bd0-8df1-1c49b161d79c-tmp-dir\") pod \"091c6742-9a7f-4bd0-8df1-1c49b161d79c\" (UID: \"091c6742-9a7f-4bd0-8df1-1c49b161d79c\") "
Sep 08 13:51:09.812946 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:09.812772    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091c6742-9a7f-4bd0-8df1-1c49b161d79c-model-cache" (OuterVolumeSpecName: "model-cache") pod "091c6742-9a7f-4bd0-8df1-1c49b161d79c" (UID: "091c6742-9a7f-4bd0-8df1-1c49b161d79c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:51:09.812946 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:09.812880    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091c6742-9a7f-4bd0-8df1-1c49b161d79c-home" (OuterVolumeSpecName: "home") pod "091c6742-9a7f-4bd0-8df1-1c49b161d79c" (UID: "091c6742-9a7f-4bd0-8df1-1c49b161d79c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:51:09.813056 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:09.813000    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091c6742-9a7f-4bd0-8df1-1c49b161d79c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "091c6742-9a7f-4bd0-8df1-1c49b161d79c" (UID: "091c6742-9a7f-4bd0-8df1-1c49b161d79c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:51:09.813056 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:09.813020    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/091c6742-9a7f-4bd0-8df1-1c49b161d79c-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:51:09.813056 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:09.813041    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/091c6742-9a7f-4bd0-8df1-1c49b161d79c-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:51:09.814911 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:09.814885    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/091c6742-9a7f-4bd0-8df1-1c49b161d79c-dshm" (OuterVolumeSpecName: "dshm") pod "091c6742-9a7f-4bd0-8df1-1c49b161d79c" (UID: "091c6742-9a7f-4bd0-8df1-1c49b161d79c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:51:09.814911 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:09.814900    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/091c6742-9a7f-4bd0-8df1-1c49b161d79c-kube-api-access-mfsz2" (OuterVolumeSpecName: "kube-api-access-mfsz2") pod "091c6742-9a7f-4bd0-8df1-1c49b161d79c" (UID: "091c6742-9a7f-4bd0-8df1-1c49b161d79c"). InnerVolumeSpecName "kube-api-access-mfsz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:51:09.815052 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:09.814919    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/091c6742-9a7f-4bd0-8df1-1c49b161d79c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "091c6742-9a7f-4bd0-8df1-1c49b161d79c" (UID: "091c6742-9a7f-4bd0-8df1-1c49b161d79c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:51:09.914179 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:09.914141    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfsz2\" (UniqueName: \"kubernetes.io/projected/091c6742-9a7f-4bd0-8df1-1c49b161d79c-kube-api-access-mfsz2\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:51:09.914179 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:09.914183    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/091c6742-9a7f-4bd0-8df1-1c49b161d79c-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:51:09.914400 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:09.914202    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/091c6742-9a7f-4bd0-8df1-1c49b161d79c-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:51:09.914400 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:09.914215    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/091c6742-9a7f-4bd0-8df1-1c49b161d79c-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:51:10.247323 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:10.247282    2558 generic.go:356] "Generic (PLEG): container finished" podID="091c6742-9a7f-4bd0-8df1-1c49b161d79c" containerID="50fd58e43fdd961d0f61bc70f3758aeea0dcb021cc68c43682b80305bd6fdeff" exitCode=0
Sep 08 13:51:10.247323 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:10.247330    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq"
Sep 08 13:51:10.247619 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:10.247372    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq" event={"ID":"091c6742-9a7f-4bd0-8df1-1c49b161d79c","Type":"ContainerDied","Data":"50fd58e43fdd961d0f61bc70f3758aeea0dcb021cc68c43682b80305bd6fdeff"}
Sep 08 13:51:10.247619 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:10.247450    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq" event={"ID":"091c6742-9a7f-4bd0-8df1-1c49b161d79c","Type":"ContainerDied","Data":"6e972416d93f07d85cb00e36381a6c7c33b9620bd95b9d44088a32108c754f3b"}
Sep 08 13:51:10.247619 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:10.247485    2558 scope.go:117] "RemoveContainer" containerID="50fd58e43fdd961d0f61bc70f3758aeea0dcb021cc68c43682b80305bd6fdeff"
Sep 08 13:51:10.257960 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:10.257937    2558 scope.go:117] "RemoveContainer" containerID="50fd58e43fdd961d0f61bc70f3758aeea0dcb021cc68c43682b80305bd6fdeff"
Sep 08 13:51:10.258295 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:51:10.258268    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50fd58e43fdd961d0f61bc70f3758aeea0dcb021cc68c43682b80305bd6fdeff\": container with ID starting with 50fd58e43fdd961d0f61bc70f3758aeea0dcb021cc68c43682b80305bd6fdeff not found: ID does not exist" containerID="50fd58e43fdd961d0f61bc70f3758aeea0dcb021cc68c43682b80305bd6fdeff"
Sep 08 13:51:10.258361 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:10.258310    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50fd58e43fdd961d0f61bc70f3758aeea0dcb021cc68c43682b80305bd6fdeff"} err="failed to get container status \"50fd58e43fdd961d0f61bc70f3758aeea0dcb021cc68c43682b80305bd6fdeff\": rpc error: code = NotFound desc = could not find container \"50fd58e43fdd961d0f61bc70f3758aeea0dcb021cc68c43682b80305bd6fdeff\": container with ID starting with 50fd58e43fdd961d0f61bc70f3758aeea0dcb021cc68c43682b80305bd6fdeff not found: ID does not exist"
Sep 08 13:51:10.280115 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:10.280083    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq"]
Sep 08 13:51:10.293006 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:10.292968    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-cq7bq"]
Sep 08 13:51:11.053663 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:11.053627    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="091c6742-9a7f-4bd0-8df1-1c49b161d79c" path="/var/lib/kubelet/pods/091c6742-9a7f-4bd0-8df1-1c49b161d79c/volumes"
Sep 08 13:51:14.910984 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:14.910935    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb"]
Sep 08 13:51:26.175469 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:26.175432    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc"
Sep 08 13:51:29.931064 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:29.930952    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb" podUID="d58c506e-6fd1-4e67-9f61-ba89882c787f" containerName="main" containerID="cri-o://486f8dc06c3a688b863ba49a998eaf83fe9122652d00661a24146511c2dea0ee" gracePeriod=45
Sep 08 13:51:29.991377 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:29.991332    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb" podUID="d58c506e-6fd1-4e67-9f61-ba89882c787f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
Sep 08 13:51:30.188614 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:30.188537    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb"
Sep 08 13:51:30.297582 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:30.297530    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d58c506e-6fd1-4e67-9f61-ba89882c787f-tmp-dir\") pod \"d58c506e-6fd1-4e67-9f61-ba89882c787f\" (UID: \"d58c506e-6fd1-4e67-9f61-ba89882c787f\") "
Sep 08 13:51:30.297582 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:30.297616    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d58c506e-6fd1-4e67-9f61-ba89882c787f-home\") pod \"d58c506e-6fd1-4e67-9f61-ba89882c787f\" (UID: \"d58c506e-6fd1-4e67-9f61-ba89882c787f\") "
Sep 08 13:51:30.297871 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:30.297644    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wf788\" (UniqueName: \"kubernetes.io/projected/d58c506e-6fd1-4e67-9f61-ba89882c787f-kube-api-access-wf788\") pod \"d58c506e-6fd1-4e67-9f61-ba89882c787f\" (UID: \"d58c506e-6fd1-4e67-9f61-ba89882c787f\") "
Sep 08 13:51:30.297871 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:30.297698    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d58c506e-6fd1-4e67-9f61-ba89882c787f-dshm\") pod \"d58c506e-6fd1-4e67-9f61-ba89882c787f\" (UID: \"d58c506e-6fd1-4e67-9f61-ba89882c787f\") "
Sep 08 13:51:30.297871 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:30.297733    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d58c506e-6fd1-4e67-9f61-ba89882c787f-model-cache\") pod \"d58c506e-6fd1-4e67-9f61-ba89882c787f\" (UID: \"d58c506e-6fd1-4e67-9f61-ba89882c787f\") "
Sep 08 13:51:30.297871 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:30.297761    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d58c506e-6fd1-4e67-9f61-ba89882c787f-tls-certs\") pod \"d58c506e-6fd1-4e67-9f61-ba89882c787f\" (UID: \"d58c506e-6fd1-4e67-9f61-ba89882c787f\") "
Sep 08 13:51:30.297871 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:30.297860    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d58c506e-6fd1-4e67-9f61-ba89882c787f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d58c506e-6fd1-4e67-9f61-ba89882c787f" (UID: "d58c506e-6fd1-4e67-9f61-ba89882c787f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:51:30.298076 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:30.297883    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d58c506e-6fd1-4e67-9f61-ba89882c787f-home" (OuterVolumeSpecName: "home") pod "d58c506e-6fd1-4e67-9f61-ba89882c787f" (UID: "d58c506e-6fd1-4e67-9f61-ba89882c787f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:51:30.298076 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:30.298011    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d58c506e-6fd1-4e67-9f61-ba89882c787f-model-cache" (OuterVolumeSpecName: "model-cache") pod "d58c506e-6fd1-4e67-9f61-ba89882c787f" (UID: "d58c506e-6fd1-4e67-9f61-ba89882c787f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:51:30.298181 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:30.298164    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d58c506e-6fd1-4e67-9f61-ba89882c787f-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:51:30.298181 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:30.298180    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d58c506e-6fd1-4e67-9f61-ba89882c787f-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:51:30.298328 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:30.298189    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d58c506e-6fd1-4e67-9f61-ba89882c787f-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:51:30.299959 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:30.299937    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d58c506e-6fd1-4e67-9f61-ba89882c787f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d58c506e-6fd1-4e67-9f61-ba89882c787f" (UID: "d58c506e-6fd1-4e67-9f61-ba89882c787f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:51:30.300036 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:30.300020    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d58c506e-6fd1-4e67-9f61-ba89882c787f-dshm" (OuterVolumeSpecName: "dshm") pod "d58c506e-6fd1-4e67-9f61-ba89882c787f" (UID: "d58c506e-6fd1-4e67-9f61-ba89882c787f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:51:30.300076 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:30.300035    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d58c506e-6fd1-4e67-9f61-ba89882c787f-kube-api-access-wf788" (OuterVolumeSpecName: "kube-api-access-wf788") pod "d58c506e-6fd1-4e67-9f61-ba89882c787f" (UID: "d58c506e-6fd1-4e67-9f61-ba89882c787f"). InnerVolumeSpecName "kube-api-access-wf788". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:51:30.349281 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:30.349241    2558 generic.go:356] "Generic (PLEG): container finished" podID="d58c506e-6fd1-4e67-9f61-ba89882c787f" containerID="486f8dc06c3a688b863ba49a998eaf83fe9122652d00661a24146511c2dea0ee" exitCode=0
Sep 08 13:51:30.349457 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:30.349293    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb"
Sep 08 13:51:30.349457 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:30.349293    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb" event={"ID":"d58c506e-6fd1-4e67-9f61-ba89882c787f","Type":"ContainerDied","Data":"486f8dc06c3a688b863ba49a998eaf83fe9122652d00661a24146511c2dea0ee"}
Sep 08 13:51:30.349457 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:30.349349    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb" event={"ID":"d58c506e-6fd1-4e67-9f61-ba89882c787f","Type":"ContainerDied","Data":"83605c5ced718951e7c7a6b8714a5afb524609c169a52f9a8718c658351987e8"}
Sep 08 13:51:30.349457 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:30.349379    2558 scope.go:117] "RemoveContainer" containerID="486f8dc06c3a688b863ba49a998eaf83fe9122652d00661a24146511c2dea0ee"
Sep 08 13:51:30.361104 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:30.361080    2558 scope.go:117] "RemoveContainer" containerID="486f8dc06c3a688b863ba49a998eaf83fe9122652d00661a24146511c2dea0ee"
Sep 08 13:51:30.361410 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:51:30.361384    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"486f8dc06c3a688b863ba49a998eaf83fe9122652d00661a24146511c2dea0ee\": container with ID starting with 486f8dc06c3a688b863ba49a998eaf83fe9122652d00661a24146511c2dea0ee not found: ID does not exist" containerID="486f8dc06c3a688b863ba49a998eaf83fe9122652d00661a24146511c2dea0ee"
Sep 08 13:51:30.361521 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:30.361419    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"486f8dc06c3a688b863ba49a998eaf83fe9122652d00661a24146511c2dea0ee"} err="failed to get container status \"486f8dc06c3a688b863ba49a998eaf83fe9122652d00661a24146511c2dea0ee\": rpc error: code = NotFound desc = could not find container \"486f8dc06c3a688b863ba49a998eaf83fe9122652d00661a24146511c2dea0ee\": container with ID starting with 486f8dc06c3a688b863ba49a998eaf83fe9122652d00661a24146511c2dea0ee not found: ID does not exist"
Sep 08 13:51:30.395404 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:30.395371    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb"]
Sep 08 13:51:30.398887 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:30.398850    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf788\" (UniqueName: \"kubernetes.io/projected/d58c506e-6fd1-4e67-9f61-ba89882c787f-kube-api-access-wf788\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:51:30.398887 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:30.398885    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d58c506e-6fd1-4e67-9f61-ba89882c787f-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:51:30.398887 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:30.398896    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d58c506e-6fd1-4e67-9f61-ba89882c787f-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:51:30.402701 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:30.402673    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-5ptdb"]
Sep 08 13:51:31.059567 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:31.059526    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d58c506e-6fd1-4e67-9f61-ba89882c787f" path="/var/lib/kubelet/pods/d58c506e-6fd1-4e67-9f61-ba89882c787f/volumes"
Sep 08 13:51:53.864372 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:51:53.864333    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc_6ac581a1-50f3-474b-995d-57c1813ae2a2/main/0.log"
Sep 08 13:52:04.743852 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:04.743816    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc"]
Sep 08 13:52:04.744392 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:04.744046    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc" podUID="6ac581a1-50f3-474b-995d-57c1813ae2a2" containerName="main" containerID="cri-o://353966e282cc95ff952294cba6da39af254a3993a264cfae909473f45f9cba49" gracePeriod=30
Sep 08 13:52:04.746586 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:04.746559    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"]
Sep 08 13:52:04.756631 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:52:04.756578    2558 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 08 13:52:04.756631 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:52:04.756631 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:52:04.756631 ip-10-0-128-147 kubenswrapper[2558]:  > containerID="42bdb65a0b3f509d07a48accc6818b92fe077710274a25ce5aa5b3e5156a77e6" cmd=["/bin/sleep","15"]
Sep 08 13:52:04.756903 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:04.756656    2558 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2" message="" err=<
Sep 08 13:52:04.756903 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:52:04.756903 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:52:04.756903 ip-10-0-128-147 kubenswrapper[2558]:  >
Sep 08 13:52:04.756903 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:52:04.756673    2558 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 08 13:52:04.756903 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:52:04.756903 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:52:04.756903 ip-10-0-128-147 kubenswrapper[2558]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2" podUID="f5560342-e146-4c6b-b5df-f94a34b13276" containerName="main" containerID="cri-o://42bdb65a0b3f509d07a48accc6818b92fe077710274a25ce5aa5b3e5156a77e6"
Sep 08 13:52:04.756903 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:04.756732    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2" podUID="f5560342-e146-4c6b-b5df-f94a34b13276" containerName="main" containerID="cri-o://42bdb65a0b3f509d07a48accc6818b92fe077710274a25ce5aa5b3e5156a77e6" gracePeriod=60
Sep 08 13:52:05.169300 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.169272    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"
Sep 08 13:52:05.300903 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.300877    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc"
Sep 08 13:52:05.315229 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.315199    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5560342-e146-4c6b-b5df-f94a34b13276-tmp-dir\") pod \"f5560342-e146-4c6b-b5df-f94a34b13276\" (UID: \"f5560342-e146-4c6b-b5df-f94a34b13276\") "
Sep 08 13:52:05.315229 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.315231    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5560342-e146-4c6b-b5df-f94a34b13276-dshm\") pod \"f5560342-e146-4c6b-b5df-f94a34b13276\" (UID: \"f5560342-e146-4c6b-b5df-f94a34b13276\") "
Sep 08 13:52:05.315436 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.315295    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5560342-e146-4c6b-b5df-f94a34b13276-kserve-provision-location\") pod \"f5560342-e146-4c6b-b5df-f94a34b13276\" (UID: \"f5560342-e146-4c6b-b5df-f94a34b13276\") "
Sep 08 13:52:05.315436 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.315314    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5560342-e146-4c6b-b5df-f94a34b13276-tls-certs\") pod \"f5560342-e146-4c6b-b5df-f94a34b13276\" (UID: \"f5560342-e146-4c6b-b5df-f94a34b13276\") "
Sep 08 13:52:05.315436 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.315337    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5560342-e146-4c6b-b5df-f94a34b13276-home\") pod \"f5560342-e146-4c6b-b5df-f94a34b13276\" (UID: \"f5560342-e146-4c6b-b5df-f94a34b13276\") "
Sep 08 13:52:05.315436 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.315369    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5560342-e146-4c6b-b5df-f94a34b13276-cabundle-cert\") pod \"f5560342-e146-4c6b-b5df-f94a34b13276\" (UID: \"f5560342-e146-4c6b-b5df-f94a34b13276\") "
Sep 08 13:52:05.315436 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.315400    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9htq\" (UniqueName: \"kubernetes.io/projected/f5560342-e146-4c6b-b5df-f94a34b13276-kube-api-access-q9htq\") pod \"f5560342-e146-4c6b-b5df-f94a34b13276\" (UID: \"f5560342-e146-4c6b-b5df-f94a34b13276\") "
Sep 08 13:52:05.315746 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.315448    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5560342-e146-4c6b-b5df-f94a34b13276-model-cache\") pod \"f5560342-e146-4c6b-b5df-f94a34b13276\" (UID: \"f5560342-e146-4c6b-b5df-f94a34b13276\") "
Sep 08 13:52:05.315746 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.315482    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5560342-e146-4c6b-b5df-f94a34b13276-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f5560342-e146-4c6b-b5df-f94a34b13276" (UID: "f5560342-e146-4c6b-b5df-f94a34b13276"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:52:05.315746 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.315618    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5560342-e146-4c6b-b5df-f94a34b13276-home" (OuterVolumeSpecName: "home") pod "f5560342-e146-4c6b-b5df-f94a34b13276" (UID: "f5560342-e146-4c6b-b5df-f94a34b13276"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:52:05.315914 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.315693    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5560342-e146-4c6b-b5df-f94a34b13276-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f5560342-e146-4c6b-b5df-f94a34b13276" (UID: "f5560342-e146-4c6b-b5df-f94a34b13276"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:52:05.315975 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.315913    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5560342-e146-4c6b-b5df-f94a34b13276-model-cache" (OuterVolumeSpecName: "model-cache") pod "f5560342-e146-4c6b-b5df-f94a34b13276" (UID: "f5560342-e146-4c6b-b5df-f94a34b13276"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:52:05.317532 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.317507    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5560342-e146-4c6b-b5df-f94a34b13276-kube-api-access-q9htq" (OuterVolumeSpecName: "kube-api-access-q9htq") pod "f5560342-e146-4c6b-b5df-f94a34b13276" (UID: "f5560342-e146-4c6b-b5df-f94a34b13276"). InnerVolumeSpecName "kube-api-access-q9htq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:52:05.317665 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.317546    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5560342-e146-4c6b-b5df-f94a34b13276-dshm" (OuterVolumeSpecName: "dshm") pod "f5560342-e146-4c6b-b5df-f94a34b13276" (UID: "f5560342-e146-4c6b-b5df-f94a34b13276"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:52:05.317665 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.317561    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5560342-e146-4c6b-b5df-f94a34b13276-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f5560342-e146-4c6b-b5df-f94a34b13276" (UID: "f5560342-e146-4c6b-b5df-f94a34b13276"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:52:05.338253 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.338203    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5560342-e146-4c6b-b5df-f94a34b13276-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f5560342-e146-4c6b-b5df-f94a34b13276" (UID: "f5560342-e146-4c6b-b5df-f94a34b13276"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:52:05.416690 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.416659    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ac581a1-50f3-474b-995d-57c1813ae2a2-cabundle-cert\") pod \"6ac581a1-50f3-474b-995d-57c1813ae2a2\" (UID: \"6ac581a1-50f3-474b-995d-57c1813ae2a2\") "
Sep 08 13:52:05.416868 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.416732    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ac581a1-50f3-474b-995d-57c1813ae2a2-tmp-dir\") pod \"6ac581a1-50f3-474b-995d-57c1813ae2a2\" (UID: \"6ac581a1-50f3-474b-995d-57c1813ae2a2\") "
Sep 08 13:52:05.416868 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.416761    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ac581a1-50f3-474b-995d-57c1813ae2a2-kserve-provision-location\") pod \"6ac581a1-50f3-474b-995d-57c1813ae2a2\" (UID: \"6ac581a1-50f3-474b-995d-57c1813ae2a2\") "
Sep 08 13:52:05.416868 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.416785    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ac581a1-50f3-474b-995d-57c1813ae2a2-tls-certs\") pod \"6ac581a1-50f3-474b-995d-57c1813ae2a2\" (UID: \"6ac581a1-50f3-474b-995d-57c1813ae2a2\") "
Sep 08 13:52:05.417036 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.417005    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ac581a1-50f3-474b-995d-57c1813ae2a2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6ac581a1-50f3-474b-995d-57c1813ae2a2" (UID: "6ac581a1-50f3-474b-995d-57c1813ae2a2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:52:05.417092 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.417055    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5560342-e146-4c6b-b5df-f94a34b13276-cabundle-cert\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:52:05.417092 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.417074    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9htq\" (UniqueName: \"kubernetes.io/projected/f5560342-e146-4c6b-b5df-f94a34b13276-kube-api-access-q9htq\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:52:05.417092 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.417088    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5560342-e146-4c6b-b5df-f94a34b13276-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:52:05.417233 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.417100    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5560342-e146-4c6b-b5df-f94a34b13276-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:52:05.417233 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.417112    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5560342-e146-4c6b-b5df-f94a34b13276-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:52:05.417233 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.417110    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac581a1-50f3-474b-995d-57c1813ae2a2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ac581a1-50f3-474b-995d-57c1813ae2a2" (UID: "6ac581a1-50f3-474b-995d-57c1813ae2a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:52:05.417233 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.417123    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5560342-e146-4c6b-b5df-f94a34b13276-kserve-provision-location\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:52:05.417233 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.417138    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5560342-e146-4c6b-b5df-f94a34b13276-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:52:05.417233 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.417149    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5560342-e146-4c6b-b5df-f94a34b13276-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:52:05.418978 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.418953    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ac581a1-50f3-474b-995d-57c1813ae2a2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ac581a1-50f3-474b-995d-57c1813ae2a2" (UID: "6ac581a1-50f3-474b-995d-57c1813ae2a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:52:05.440028 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.439991    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac581a1-50f3-474b-995d-57c1813ae2a2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6ac581a1-50f3-474b-995d-57c1813ae2a2" (UID: "6ac581a1-50f3-474b-995d-57c1813ae2a2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:52:05.518567 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.518531    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ac581a1-50f3-474b-995d-57c1813ae2a2-cabundle-cert\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:52:05.518567 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.518561    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ac581a1-50f3-474b-995d-57c1813ae2a2-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:52:05.518801 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.518577    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ac581a1-50f3-474b-995d-57c1813ae2a2-kserve-provision-location\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:52:05.518801 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.518620    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ac581a1-50f3-474b-995d-57c1813ae2a2-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:52:05.527361 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.527325    2558 generic.go:356] "Generic (PLEG): container finished" podID="f5560342-e146-4c6b-b5df-f94a34b13276" containerID="42bdb65a0b3f509d07a48accc6818b92fe077710274a25ce5aa5b3e5156a77e6" exitCode=0
Sep 08 13:52:05.527526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.527381    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"
Sep 08 13:52:05.527526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.527419    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2" event={"ID":"f5560342-e146-4c6b-b5df-f94a34b13276","Type":"ContainerDied","Data":"42bdb65a0b3f509d07a48accc6818b92fe077710274a25ce5aa5b3e5156a77e6"}
Sep 08 13:52:05.527526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.527472    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2" event={"ID":"f5560342-e146-4c6b-b5df-f94a34b13276","Type":"ContainerDied","Data":"9b59fc20d66db85c1533bd0540460130b8a320b298a04583a287a1d12c96a9fc"}
Sep 08 13:52:05.527526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.527497    2558 scope.go:117] "RemoveContainer" containerID="42bdb65a0b3f509d07a48accc6818b92fe077710274a25ce5aa5b3e5156a77e6"
Sep 08 13:52:05.529247 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.529226    2558 generic.go:356] "Generic (PLEG): container finished" podID="6ac581a1-50f3-474b-995d-57c1813ae2a2" containerID="353966e282cc95ff952294cba6da39af254a3993a264cfae909473f45f9cba49" exitCode=0
Sep 08 13:52:05.529358 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.529277    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc"
Sep 08 13:52:05.529358 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.529282    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc" event={"ID":"6ac581a1-50f3-474b-995d-57c1813ae2a2","Type":"ContainerDied","Data":"353966e282cc95ff952294cba6da39af254a3993a264cfae909473f45f9cba49"}
Sep 08 13:52:05.529358 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.529318    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc" event={"ID":"6ac581a1-50f3-474b-995d-57c1813ae2a2","Type":"ContainerDied","Data":"d542e9bebd23da3b31bde70b7e00d9e18ec6f717a212e082e3890ed9718d5906"}
Sep 08 13:52:05.538184 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.538158    2558 scope.go:117] "RemoveContainer" containerID="9ec66f0227d72da98c2ea8b026b623e975f940066409a2b6dc247b9e1253a249"
Sep 08 13:52:05.546520 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.546501    2558 scope.go:117] "RemoveContainer" containerID="42bdb65a0b3f509d07a48accc6818b92fe077710274a25ce5aa5b3e5156a77e6"
Sep 08 13:52:05.546824 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:52:05.546804    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42bdb65a0b3f509d07a48accc6818b92fe077710274a25ce5aa5b3e5156a77e6\": container with ID starting with 42bdb65a0b3f509d07a48accc6818b92fe077710274a25ce5aa5b3e5156a77e6 not found: ID does not exist" containerID="42bdb65a0b3f509d07a48accc6818b92fe077710274a25ce5aa5b3e5156a77e6"
Sep 08 13:52:05.546899 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.546830    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42bdb65a0b3f509d07a48accc6818b92fe077710274a25ce5aa5b3e5156a77e6"} err="failed to get container status \"42bdb65a0b3f509d07a48accc6818b92fe077710274a25ce5aa5b3e5156a77e6\": rpc error: code = NotFound desc = could not find container \"42bdb65a0b3f509d07a48accc6818b92fe077710274a25ce5aa5b3e5156a77e6\": container with ID starting with 42bdb65a0b3f509d07a48accc6818b92fe077710274a25ce5aa5b3e5156a77e6 not found: ID does not exist"
Sep 08 13:52:05.546899 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.546847    2558 scope.go:117] "RemoveContainer" containerID="9ec66f0227d72da98c2ea8b026b623e975f940066409a2b6dc247b9e1253a249"
Sep 08 13:52:05.547111 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:52:05.547094    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ec66f0227d72da98c2ea8b026b623e975f940066409a2b6dc247b9e1253a249\": container with ID starting with 9ec66f0227d72da98c2ea8b026b623e975f940066409a2b6dc247b9e1253a249 not found: ID does not exist" containerID="9ec66f0227d72da98c2ea8b026b623e975f940066409a2b6dc247b9e1253a249"
Sep 08 13:52:05.547156 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.547117    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ec66f0227d72da98c2ea8b026b623e975f940066409a2b6dc247b9e1253a249"} err="failed to get container status \"9ec66f0227d72da98c2ea8b026b623e975f940066409a2b6dc247b9e1253a249\": rpc error: code = NotFound desc = could not find container \"9ec66f0227d72da98c2ea8b026b623e975f940066409a2b6dc247b9e1253a249\": container with ID starting with 9ec66f0227d72da98c2ea8b026b623e975f940066409a2b6dc247b9e1253a249 not found: ID does not exist"
Sep 08 13:52:05.547156 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.547133    2558 scope.go:117] "RemoveContainer" containerID="353966e282cc95ff952294cba6da39af254a3993a264cfae909473f45f9cba49"
Sep 08 13:52:05.556182 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.556031    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"]
Sep 08 13:52:05.557359 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.557341    2558 scope.go:117] "RemoveContainer" containerID="bf3881d3e62c3be6246cf03e200286b97f73e32d1cc0831fa702a0a7adc258c5"
Sep 08 13:52:05.563398 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.563373    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-5897d7c8b8-ljgq2"]
Sep 08 13:52:05.566541 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.566518    2558 scope.go:117] "RemoveContainer" containerID="353966e282cc95ff952294cba6da39af254a3993a264cfae909473f45f9cba49"
Sep 08 13:52:05.566949 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:52:05.566922    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"353966e282cc95ff952294cba6da39af254a3993a264cfae909473f45f9cba49\": container with ID starting with 353966e282cc95ff952294cba6da39af254a3993a264cfae909473f45f9cba49 not found: ID does not exist" containerID="353966e282cc95ff952294cba6da39af254a3993a264cfae909473f45f9cba49"
Sep 08 13:52:05.567019 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.566957    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"353966e282cc95ff952294cba6da39af254a3993a264cfae909473f45f9cba49"} err="failed to get container status \"353966e282cc95ff952294cba6da39af254a3993a264cfae909473f45f9cba49\": rpc error: code = NotFound desc = could not find container \"353966e282cc95ff952294cba6da39af254a3993a264cfae909473f45f9cba49\": container with ID starting with 353966e282cc95ff952294cba6da39af254a3993a264cfae909473f45f9cba49 not found: ID does not exist"
Sep 08 13:52:05.567019 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.566974    2558 scope.go:117] "RemoveContainer" containerID="bf3881d3e62c3be6246cf03e200286b97f73e32d1cc0831fa702a0a7adc258c5"
Sep 08 13:52:05.567225 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:52:05.567209    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf3881d3e62c3be6246cf03e200286b97f73e32d1cc0831fa702a0a7adc258c5\": container with ID starting with bf3881d3e62c3be6246cf03e200286b97f73e32d1cc0831fa702a0a7adc258c5 not found: ID does not exist" containerID="bf3881d3e62c3be6246cf03e200286b97f73e32d1cc0831fa702a0a7adc258c5"
Sep 08 13:52:05.567263 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.567231    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf3881d3e62c3be6246cf03e200286b97f73e32d1cc0831fa702a0a7adc258c5"} err="failed to get container status \"bf3881d3e62c3be6246cf03e200286b97f73e32d1cc0831fa702a0a7adc258c5\": rpc error: code = NotFound desc = could not find container \"bf3881d3e62c3be6246cf03e200286b97f73e32d1cc0831fa702a0a7adc258c5\": container with ID starting with bf3881d3e62c3be6246cf03e200286b97f73e32d1cc0831fa702a0a7adc258c5 not found: ID does not exist"
Sep 08 13:52:05.574713 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.574693    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc"]
Sep 08 13:52:05.578749 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:05.578729    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-79ddb54454-ss8jc"]
Sep 08 13:52:07.054953 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:07.054911    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ac581a1-50f3-474b-995d-57c1813ae2a2" path="/var/lib/kubelet/pods/6ac581a1-50f3-474b-995d-57c1813ae2a2/volumes"
Sep 08 13:52:07.055548 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:07.055527    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5560342-e146-4c6b-b5df-f94a34b13276" path="/var/lib/kubelet/pods/f5560342-e146-4c6b-b5df-f94a34b13276/volumes"
Sep 08 13:52:18.859223 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:18.859185    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q"]
Sep 08 13:52:18.859656 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:18.859551    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="091c6742-9a7f-4bd0-8df1-1c49b161d79c" containerName="main"
Sep 08 13:52:18.859656 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:18.859563    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="091c6742-9a7f-4bd0-8df1-1c49b161d79c" containerName="main"
Sep 08 13:52:18.859656 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:18.859572    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ac581a1-50f3-474b-995d-57c1813ae2a2" containerName="storage-initializer"
Sep 08 13:52:18.859656 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:18.859577    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ac581a1-50f3-474b-995d-57c1813ae2a2" containerName="storage-initializer"
Sep 08 13:52:18.859656 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:18.859614    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5560342-e146-4c6b-b5df-f94a34b13276" containerName="main"
Sep 08 13:52:18.859656 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:18.859632    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5560342-e146-4c6b-b5df-f94a34b13276" containerName="main"
Sep 08 13:52:18.859656 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:18.859639    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d58c506e-6fd1-4e67-9f61-ba89882c787f" containerName="main"
Sep 08 13:52:18.859656 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:18.859644    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d58c506e-6fd1-4e67-9f61-ba89882c787f" containerName="main"
Sep 08 13:52:18.859656 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:18.859651    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5560342-e146-4c6b-b5df-f94a34b13276" containerName="storage-initializer"
Sep 08 13:52:18.859656 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:18.859656    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5560342-e146-4c6b-b5df-f94a34b13276" containerName="storage-initializer"
Sep 08 13:52:18.859656 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:18.859670    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ac581a1-50f3-474b-995d-57c1813ae2a2" containerName="main"
Sep 08 13:52:18.860304 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:18.859675    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ac581a1-50f3-474b-995d-57c1813ae2a2" containerName="main"
Sep 08 13:52:18.860304 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:18.859740    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5560342-e146-4c6b-b5df-f94a34b13276" containerName="main"
Sep 08 13:52:18.860304 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:18.859746    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ac581a1-50f3-474b-995d-57c1813ae2a2" containerName="main"
Sep 08 13:52:18.860304 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:18.859753    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d58c506e-6fd1-4e67-9f61-ba89882c787f" containerName="main"
Sep 08 13:52:18.860304 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:18.859757    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="091c6742-9a7f-4bd0-8df1-1c49b161d79c" containerName="main"
Sep 08 13:52:18.862726 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:18.862709    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q"
Sep 08 13:52:18.865181 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:18.865161    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"kube-root-ca.crt\""
Sep 08 13:52:18.866513 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:18.866484    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"default-dockercfg-bnzrw\""
Sep 08 13:52:18.866742 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:18.866723    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"openshift-service-ca.crt\""
Sep 08 13:52:18.866947 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:18.866930    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v1-kserve-self-signed-certs\""
Sep 08 13:52:18.903640 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:18.903607    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q"]
Sep 08 13:52:19.038852 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:19.038812    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a180d75f-2f98-4045-b642-7acd13bc94d4-home\") pod \"tri-v1-kserve-79f8f98bc5-wlk5q\" (UID: \"a180d75f-2f98-4045-b642-7acd13bc94d4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q"
Sep 08 13:52:19.038852 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:19.038855    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a180d75f-2f98-4045-b642-7acd13bc94d4-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-wlk5q\" (UID: \"a180d75f-2f98-4045-b642-7acd13bc94d4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q"
Sep 08 13:52:19.039096 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:19.038918    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a180d75f-2f98-4045-b642-7acd13bc94d4-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-wlk5q\" (UID: \"a180d75f-2f98-4045-b642-7acd13bc94d4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q"
Sep 08 13:52:19.039096 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:19.038980    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a180d75f-2f98-4045-b642-7acd13bc94d4-dshm\") pod \"tri-v1-kserve-79f8f98bc5-wlk5q\" (UID: \"a180d75f-2f98-4045-b642-7acd13bc94d4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q"
Sep 08 13:52:19.039096 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:19.039006    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a180d75f-2f98-4045-b642-7acd13bc94d4-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-wlk5q\" (UID: \"a180d75f-2f98-4045-b642-7acd13bc94d4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q"
Sep 08 13:52:19.039096 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:19.039036    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcmwt\" (UniqueName: \"kubernetes.io/projected/a180d75f-2f98-4045-b642-7acd13bc94d4-kube-api-access-vcmwt\") pod \"tri-v1-kserve-79f8f98bc5-wlk5q\" (UID: \"a180d75f-2f98-4045-b642-7acd13bc94d4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q"
Sep 08 13:52:19.139844 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:19.139742    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a180d75f-2f98-4045-b642-7acd13bc94d4-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-wlk5q\" (UID: \"a180d75f-2f98-4045-b642-7acd13bc94d4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q"
Sep 08 13:52:19.139844 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:19.139792    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a180d75f-2f98-4045-b642-7acd13bc94d4-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-wlk5q\" (UID: \"a180d75f-2f98-4045-b642-7acd13bc94d4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q"
Sep 08 13:52:19.139844 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:19.139829    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a180d75f-2f98-4045-b642-7acd13bc94d4-dshm\") pod \"tri-v1-kserve-79f8f98bc5-wlk5q\" (UID: \"a180d75f-2f98-4045-b642-7acd13bc94d4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q"
Sep 08 13:52:19.140109 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:19.139853    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a180d75f-2f98-4045-b642-7acd13bc94d4-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-wlk5q\" (UID: \"a180d75f-2f98-4045-b642-7acd13bc94d4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q"
Sep 08 13:52:19.140109 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:19.139874    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcmwt\" (UniqueName: \"kubernetes.io/projected/a180d75f-2f98-4045-b642-7acd13bc94d4-kube-api-access-vcmwt\") pod \"tri-v1-kserve-79f8f98bc5-wlk5q\" (UID: \"a180d75f-2f98-4045-b642-7acd13bc94d4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q"
Sep 08 13:52:19.140109 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:19.139931    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a180d75f-2f98-4045-b642-7acd13bc94d4-home\") pod \"tri-v1-kserve-79f8f98bc5-wlk5q\" (UID: \"a180d75f-2f98-4045-b642-7acd13bc94d4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q"
Sep 08 13:52:19.140277 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:19.140206    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a180d75f-2f98-4045-b642-7acd13bc94d4-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-wlk5q\" (UID: \"a180d75f-2f98-4045-b642-7acd13bc94d4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q"
Sep 08 13:52:19.140277 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:19.140258    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a180d75f-2f98-4045-b642-7acd13bc94d4-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-wlk5q\" (UID: \"a180d75f-2f98-4045-b642-7acd13bc94d4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q"
Sep 08 13:52:19.140374 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:19.140287    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a180d75f-2f98-4045-b642-7acd13bc94d4-home\") pod \"tri-v1-kserve-79f8f98bc5-wlk5q\" (UID: \"a180d75f-2f98-4045-b642-7acd13bc94d4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q"
Sep 08 13:52:19.142307 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:19.142280    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a180d75f-2f98-4045-b642-7acd13bc94d4-dshm\") pod \"tri-v1-kserve-79f8f98bc5-wlk5q\" (UID: \"a180d75f-2f98-4045-b642-7acd13bc94d4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q"
Sep 08 13:52:19.142510 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:19.142490    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a180d75f-2f98-4045-b642-7acd13bc94d4-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-wlk5q\" (UID: \"a180d75f-2f98-4045-b642-7acd13bc94d4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q"
Sep 08 13:52:19.149052 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:19.149025    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcmwt\" (UniqueName: \"kubernetes.io/projected/a180d75f-2f98-4045-b642-7acd13bc94d4-kube-api-access-vcmwt\") pod \"tri-v1-kserve-79f8f98bc5-wlk5q\" (UID: \"a180d75f-2f98-4045-b642-7acd13bc94d4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q"
Sep 08 13:52:19.172856 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:19.172809    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q"
Sep 08 13:52:19.457910 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:19.457881    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q"]
Sep 08 13:52:19.460651 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:52:19.460617    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda180d75f_2f98_4045_b642_7acd13bc94d4.slice/crio-2e83c9cb5203dd1bf8ca5a6a00864580b50c34fa7065f929edd8e3e3edb0decc WatchSource:0}: Error finding container 2e83c9cb5203dd1bf8ca5a6a00864580b50c34fa7065f929edd8e3e3edb0decc: Status 404 returned error can't find the container with id 2e83c9cb5203dd1bf8ca5a6a00864580b50c34fa7065f929edd8e3e3edb0decc
Sep 08 13:52:19.598219 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:19.598181    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q" event={"ID":"a180d75f-2f98-4045-b642-7acd13bc94d4","Type":"ContainerStarted","Data":"7d9ef70df098973338be598021f318afa563877258454661ca0b65f1beea631e"}
Sep 08 13:52:19.598413 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:19.598232    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q" event={"ID":"a180d75f-2f98-4045-b642-7acd13bc94d4","Type":"ContainerStarted","Data":"2e83c9cb5203dd1bf8ca5a6a00864580b50c34fa7065f929edd8e3e3edb0decc"}
Sep 08 13:52:19.634204 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:19.634134    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q" podStartSLOduration=1.634109886 podStartE2EDuration="1.634109886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:52:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:52:19.632916862 +0000 UTC m=+2203.084677794" watchObservedRunningTime="2026-09-08 13:52:19.634109886 +0000 UTC m=+2203.085870819"
Sep 08 13:52:20.173810 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:20.173772    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q"
Sep 08 13:52:28.302019 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:28.301980    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm"]
Sep 08 13:52:28.311099 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:28.311073    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm"
Sep 08 13:52:28.313558 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:28.313534    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v2-kserve-self-signed-certs\""
Sep 08 13:52:28.316524 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:28.316499    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9864420c-5be6-442d-9381-79db08992db3-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-pctqm\" (UID: \"9864420c-5be6-442d-9381-79db08992db3\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm"
Sep 08 13:52:28.316670 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:28.316544    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlzw5\" (UniqueName: \"kubernetes.io/projected/9864420c-5be6-442d-9381-79db08992db3-kube-api-access-qlzw5\") pod \"tri-v2-kserve-dcb554bf5-pctqm\" (UID: \"9864420c-5be6-442d-9381-79db08992db3\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm"
Sep 08 13:52:28.316670 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:28.316580    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9864420c-5be6-442d-9381-79db08992db3-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-pctqm\" (UID: \"9864420c-5be6-442d-9381-79db08992db3\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm"
Sep 08 13:52:28.316670 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:28.316644    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9864420c-5be6-442d-9381-79db08992db3-dshm\") pod \"tri-v2-kserve-dcb554bf5-pctqm\" (UID: \"9864420c-5be6-442d-9381-79db08992db3\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm"
Sep 08 13:52:28.316854 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:28.316676    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9864420c-5be6-442d-9381-79db08992db3-home\") pod \"tri-v2-kserve-dcb554bf5-pctqm\" (UID: \"9864420c-5be6-442d-9381-79db08992db3\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm"
Sep 08 13:52:28.316854 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:28.316774    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9864420c-5be6-442d-9381-79db08992db3-model-cache\") pod \"tri-v2-kserve-dcb554bf5-pctqm\" (UID: \"9864420c-5be6-442d-9381-79db08992db3\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm"
Sep 08 13:52:28.318066 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:28.318041    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm"]
Sep 08 13:52:28.418159 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:28.418122    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9864420c-5be6-442d-9381-79db08992db3-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-pctqm\" (UID: \"9864420c-5be6-442d-9381-79db08992db3\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm"
Sep 08 13:52:28.418353 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:28.418167    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlzw5\" (UniqueName: \"kubernetes.io/projected/9864420c-5be6-442d-9381-79db08992db3-kube-api-access-qlzw5\") pod \"tri-v2-kserve-dcb554bf5-pctqm\" (UID: \"9864420c-5be6-442d-9381-79db08992db3\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm"
Sep 08 13:52:28.418353 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:28.418293    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9864420c-5be6-442d-9381-79db08992db3-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-pctqm\" (UID: \"9864420c-5be6-442d-9381-79db08992db3\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm"
Sep 08 13:52:28.418436 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:28.418361    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9864420c-5be6-442d-9381-79db08992db3-dshm\") pod \"tri-v2-kserve-dcb554bf5-pctqm\" (UID: \"9864420c-5be6-442d-9381-79db08992db3\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm"
Sep 08 13:52:28.418436 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:28.418413    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9864420c-5be6-442d-9381-79db08992db3-home\") pod \"tri-v2-kserve-dcb554bf5-pctqm\" (UID: \"9864420c-5be6-442d-9381-79db08992db3\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm"
Sep 08 13:52:28.418509 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:28.418458    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9864420c-5be6-442d-9381-79db08992db3-model-cache\") pod \"tri-v2-kserve-dcb554bf5-pctqm\" (UID: \"9864420c-5be6-442d-9381-79db08992db3\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm"
Sep 08 13:52:28.418751 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:28.418726    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9864420c-5be6-442d-9381-79db08992db3-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-pctqm\" (UID: \"9864420c-5be6-442d-9381-79db08992db3\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm"
Sep 08 13:52:28.418907 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:28.418797    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9864420c-5be6-442d-9381-79db08992db3-home\") pod \"tri-v2-kserve-dcb554bf5-pctqm\" (UID: \"9864420c-5be6-442d-9381-79db08992db3\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm"
Sep 08 13:52:28.418907 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:28.418842    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9864420c-5be6-442d-9381-79db08992db3-model-cache\") pod \"tri-v2-kserve-dcb554bf5-pctqm\" (UID: \"9864420c-5be6-442d-9381-79db08992db3\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm"
Sep 08 13:52:28.420635 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:28.420615    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9864420c-5be6-442d-9381-79db08992db3-dshm\") pod \"tri-v2-kserve-dcb554bf5-pctqm\" (UID: \"9864420c-5be6-442d-9381-79db08992db3\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm"
Sep 08 13:52:28.420833 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:28.420808    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9864420c-5be6-442d-9381-79db08992db3-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-pctqm\" (UID: \"9864420c-5be6-442d-9381-79db08992db3\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm"
Sep 08 13:52:28.427040 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:28.427017    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlzw5\" (UniqueName: \"kubernetes.io/projected/9864420c-5be6-442d-9381-79db08992db3-kube-api-access-qlzw5\") pod \"tri-v2-kserve-dcb554bf5-pctqm\" (UID: \"9864420c-5be6-442d-9381-79db08992db3\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm"
Sep 08 13:52:28.623604 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:28.623560    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm"
Sep 08 13:52:28.903348 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:28.903225    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm"]
Sep 08 13:52:28.906155 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:52:28.906118    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9864420c_5be6_442d_9381_79db08992db3.slice/crio-1d085cf9c3c3e6bb34705468820329ee9023e65cb669830def33609747265bdc WatchSource:0}: Error finding container 1d085cf9c3c3e6bb34705468820329ee9023e65cb669830def33609747265bdc: Status 404 returned error can't find the container with id 1d085cf9c3c3e6bb34705468820329ee9023e65cb669830def33609747265bdc
Sep 08 13:52:29.173791 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:29.173700    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q"
Sep 08 13:52:29.187103 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:29.187070    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q"
Sep 08 13:52:29.654166 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:29.654132    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm" event={"ID":"9864420c-5be6-442d-9381-79db08992db3","Type":"ContainerStarted","Data":"2ed645a0d66f87a2ac7a3c59241e73cca725abde9d3de63b4c6c8184ff3a4e5a"}
Sep 08 13:52:29.654630 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:29.654188    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm" event={"ID":"9864420c-5be6-442d-9381-79db08992db3","Type":"ContainerStarted","Data":"1d085cf9c3c3e6bb34705468820329ee9023e65cb669830def33609747265bdc"}
Sep 08 13:52:29.665460 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:29.665435    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q"
Sep 08 13:52:29.672681 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:29.672634    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm" podStartSLOduration=1.672617925 podStartE2EDuration="1.672617925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:52:29.671671043 +0000 UTC m=+2213.123431974" watchObservedRunningTime="2026-09-08 13:52:29.672617925 +0000 UTC m=+2213.124378896"
Sep 08 13:52:30.624233 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:30.624183    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm"
Sep 08 13:52:34.413110 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:34.413070    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl"]
Sep 08 13:52:34.419361 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:34.419330    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl"
Sep 08 13:52:34.423490 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:34.423462    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v3-kserve-self-signed-certs\""
Sep 08 13:52:34.431543 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:34.431513    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl"]
Sep 08 13:52:34.466715 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:34.466670    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-dshm\") pod \"tri-v3-kserve-798c88c66c-fgzjl\" (UID: \"92d05aef-6bc0-47ae-8abc-ab11a6416c2d\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl"
Sep 08 13:52:34.466715 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:34.466713    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-home\") pod \"tri-v3-kserve-798c88c66c-fgzjl\" (UID: \"92d05aef-6bc0-47ae-8abc-ab11a6416c2d\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl"
Sep 08 13:52:34.466933 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:34.466738    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-tls-certs\") pod \"tri-v3-kserve-798c88c66c-fgzjl\" (UID: \"92d05aef-6bc0-47ae-8abc-ab11a6416c2d\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl"
Sep 08 13:52:34.466933 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:34.466806    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-model-cache\") pod \"tri-v3-kserve-798c88c66c-fgzjl\" (UID: \"92d05aef-6bc0-47ae-8abc-ab11a6416c2d\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl"
Sep 08 13:52:34.466933 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:34.466852    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggzdp\" (UniqueName: \"kubernetes.io/projected/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-kube-api-access-ggzdp\") pod \"tri-v3-kserve-798c88c66c-fgzjl\" (UID: \"92d05aef-6bc0-47ae-8abc-ab11a6416c2d\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl"
Sep 08 13:52:34.466933 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:34.466890    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-fgzjl\" (UID: \"92d05aef-6bc0-47ae-8abc-ab11a6416c2d\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl"
Sep 08 13:52:34.567406 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:34.567369    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-fgzjl\" (UID: \"92d05aef-6bc0-47ae-8abc-ab11a6416c2d\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl"
Sep 08 13:52:34.567616 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:34.567451    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-dshm\") pod \"tri-v3-kserve-798c88c66c-fgzjl\" (UID: \"92d05aef-6bc0-47ae-8abc-ab11a6416c2d\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl"
Sep 08 13:52:34.567616 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:34.567473    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-home\") pod \"tri-v3-kserve-798c88c66c-fgzjl\" (UID: \"92d05aef-6bc0-47ae-8abc-ab11a6416c2d\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl"
Sep 08 13:52:34.567616 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:34.567493    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-tls-certs\") pod \"tri-v3-kserve-798c88c66c-fgzjl\" (UID: \"92d05aef-6bc0-47ae-8abc-ab11a6416c2d\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl"
Sep 08 13:52:34.567616 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:34.567523    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-model-cache\") pod \"tri-v3-kserve-798c88c66c-fgzjl\" (UID: \"92d05aef-6bc0-47ae-8abc-ab11a6416c2d\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl"
Sep 08 13:52:34.567616 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:34.567566    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggzdp\" (UniqueName: \"kubernetes.io/projected/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-kube-api-access-ggzdp\") pod \"tri-v3-kserve-798c88c66c-fgzjl\" (UID: \"92d05aef-6bc0-47ae-8abc-ab11a6416c2d\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl"
Sep 08 13:52:34.567892 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:34.567839    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-fgzjl\" (UID: \"92d05aef-6bc0-47ae-8abc-ab11a6416c2d\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl"
Sep 08 13:52:34.567944 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:34.567914    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-model-cache\") pod \"tri-v3-kserve-798c88c66c-fgzjl\" (UID: \"92d05aef-6bc0-47ae-8abc-ab11a6416c2d\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl"
Sep 08 13:52:34.567993 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:34.567975    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-home\") pod \"tri-v3-kserve-798c88c66c-fgzjl\" (UID: \"92d05aef-6bc0-47ae-8abc-ab11a6416c2d\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl"
Sep 08 13:52:34.571030 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:34.569966    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-dshm\") pod \"tri-v3-kserve-798c88c66c-fgzjl\" (UID: \"92d05aef-6bc0-47ae-8abc-ab11a6416c2d\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl"
Sep 08 13:52:34.571030 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:34.570311    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-tls-certs\") pod \"tri-v3-kserve-798c88c66c-fgzjl\" (UID: \"92d05aef-6bc0-47ae-8abc-ab11a6416c2d\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl"
Sep 08 13:52:34.581101 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:34.581071    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggzdp\" (UniqueName: \"kubernetes.io/projected/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-kube-api-access-ggzdp\") pod \"tri-v3-kserve-798c88c66c-fgzjl\" (UID: \"92d05aef-6bc0-47ae-8abc-ab11a6416c2d\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl"
Sep 08 13:52:34.735362 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:34.735265    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl"
Sep 08 13:52:35.009767 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:35.009744    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl"]
Sep 08 13:52:35.012250 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:52:35.012220    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d05aef_6bc0_47ae_8abc_ab11a6416c2d.slice/crio-ffd36c93000a30321ce42c711055281c7eb3a4d23ec02e6dfd20b37fd922f40f WatchSource:0}: Error finding container ffd36c93000a30321ce42c711055281c7eb3a4d23ec02e6dfd20b37fd922f40f: Status 404 returned error can't find the container with id ffd36c93000a30321ce42c711055281c7eb3a4d23ec02e6dfd20b37fd922f40f
Sep 08 13:52:35.683187 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:35.683149    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl" event={"ID":"92d05aef-6bc0-47ae-8abc-ab11a6416c2d","Type":"ContainerStarted","Data":"47ba65954c50a24005a910f9cdd5236057591369b959a24a010217c5f7f20ca8"}
Sep 08 13:52:35.683660 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:35.683215    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl" event={"ID":"92d05aef-6bc0-47ae-8abc-ab11a6416c2d","Type":"ContainerStarted","Data":"ffd36c93000a30321ce42c711055281c7eb3a4d23ec02e6dfd20b37fd922f40f"}
Sep 08 13:52:35.701888 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:35.701834    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl" podStartSLOduration=1.7018153489999999 podStartE2EDuration="1.701815349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:52:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:52:35.701208371 +0000 UTC m=+2219.152969302" watchObservedRunningTime="2026-09-08 13:52:35.701815349 +0000 UTC m=+2219.153576281"
Sep 08 13:52:35.735652 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:35.735584    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl"
Sep 08 13:52:38.624190 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:38.624145    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm"
Sep 08 13:52:38.636942 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:38.636915    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm"
Sep 08 13:52:38.708694 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:38.708663    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm"
Sep 08 13:52:44.735854 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:44.735819    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl"
Sep 08 13:52:44.748230 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:44.748195    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl"
Sep 08 13:52:45.743088 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:52:45.743057    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl"
Sep 08 13:53:37.330750 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.330708    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"]
Sep 08 13:53:37.334819 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.334797    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"
Sep 08 13:53:37.338380 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.338358    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 08 13:53:37.338526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.338386    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 08 13:53:37.338526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.338408    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:53:37.339233 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.339212    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 08 13:53:37.339320 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.339289    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-pwm66\""
Sep 08 13:53:37.339383 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.339289    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 08 13:53:37.363095 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.363068    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"]
Sep 08 13:53:37.445065 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.445021    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwcrv\" (UniqueName: \"kubernetes.io/projected/006c079d-a72d-40c3-8e75-df1349bd44d4-kube-api-access-fwcrv\") pod \"tokenizer-template-args-test-kserve-57ff94957b-h49l4\" (UID: \"006c079d-a72d-40c3-8e75-df1349bd44d4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"
Sep 08 13:53:37.445065 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.445066    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/006c079d-a72d-40c3-8e75-df1349bd44d4-model-cache\") pod \"tokenizer-template-args-test-kserve-57ff94957b-h49l4\" (UID: \"006c079d-a72d-40c3-8e75-df1349bd44d4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"
Sep 08 13:53:37.445329 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.445162    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/006c079d-a72d-40c3-8e75-df1349bd44d4-home\") pod \"tokenizer-template-args-test-kserve-57ff94957b-h49l4\" (UID: \"006c079d-a72d-40c3-8e75-df1349bd44d4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"
Sep 08 13:53:37.445329 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.445215    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/006c079d-a72d-40c3-8e75-df1349bd44d4-dshm\") pod \"tokenizer-template-args-test-kserve-57ff94957b-h49l4\" (UID: \"006c079d-a72d-40c3-8e75-df1349bd44d4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"
Sep 08 13:53:37.445329 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.445238    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/006c079d-a72d-40c3-8e75-df1349bd44d4-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-57ff94957b-h49l4\" (UID: \"006c079d-a72d-40c3-8e75-df1349bd44d4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"
Sep 08 13:53:37.445329 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.445269    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/006c079d-a72d-40c3-8e75-df1349bd44d4-tls-certs\") pod \"tokenizer-template-args-test-kserve-57ff94957b-h49l4\" (UID: \"006c079d-a72d-40c3-8e75-df1349bd44d4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"
Sep 08 13:53:37.445494 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.445339    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/006c079d-a72d-40c3-8e75-df1349bd44d4-tmp-dir\") pod \"tokenizer-template-args-test-kserve-57ff94957b-h49l4\" (UID: \"006c079d-a72d-40c3-8e75-df1349bd44d4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"
Sep 08 13:53:37.445494 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.445378    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/006c079d-a72d-40c3-8e75-df1349bd44d4-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-57ff94957b-h49l4\" (UID: \"006c079d-a72d-40c3-8e75-df1349bd44d4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"
Sep 08 13:53:37.545989 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.545954    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/006c079d-a72d-40c3-8e75-df1349bd44d4-tmp-dir\") pod \"tokenizer-template-args-test-kserve-57ff94957b-h49l4\" (UID: \"006c079d-a72d-40c3-8e75-df1349bd44d4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"
Sep 08 13:53:37.546176 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.546007    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/006c079d-a72d-40c3-8e75-df1349bd44d4-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-57ff94957b-h49l4\" (UID: \"006c079d-a72d-40c3-8e75-df1349bd44d4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"
Sep 08 13:53:37.546176 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.546054    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwcrv\" (UniqueName: \"kubernetes.io/projected/006c079d-a72d-40c3-8e75-df1349bd44d4-kube-api-access-fwcrv\") pod \"tokenizer-template-args-test-kserve-57ff94957b-h49l4\" (UID: \"006c079d-a72d-40c3-8e75-df1349bd44d4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"
Sep 08 13:53:37.546176 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.546080    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/006c079d-a72d-40c3-8e75-df1349bd44d4-model-cache\") pod \"tokenizer-template-args-test-kserve-57ff94957b-h49l4\" (UID: \"006c079d-a72d-40c3-8e75-df1349bd44d4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"
Sep 08 13:53:37.546344 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.546234    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/006c079d-a72d-40c3-8e75-df1349bd44d4-home\") pod \"tokenizer-template-args-test-kserve-57ff94957b-h49l4\" (UID: \"006c079d-a72d-40c3-8e75-df1349bd44d4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"
Sep 08 13:53:37.546344 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.546294    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/006c079d-a72d-40c3-8e75-df1349bd44d4-dshm\") pod \"tokenizer-template-args-test-kserve-57ff94957b-h49l4\" (UID: \"006c079d-a72d-40c3-8e75-df1349bd44d4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"
Sep 08 13:53:37.546454 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.546345    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/006c079d-a72d-40c3-8e75-df1349bd44d4-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-57ff94957b-h49l4\" (UID: \"006c079d-a72d-40c3-8e75-df1349bd44d4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"
Sep 08 13:53:37.546454 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.546378    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/006c079d-a72d-40c3-8e75-df1349bd44d4-tmp-dir\") pod \"tokenizer-template-args-test-kserve-57ff94957b-h49l4\" (UID: \"006c079d-a72d-40c3-8e75-df1349bd44d4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"
Sep 08 13:53:37.546454 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.546446    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/006c079d-a72d-40c3-8e75-df1349bd44d4-model-cache\") pod \"tokenizer-template-args-test-kserve-57ff94957b-h49l4\" (UID: \"006c079d-a72d-40c3-8e75-df1349bd44d4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"
Sep 08 13:53:37.546655 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.546487    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/006c079d-a72d-40c3-8e75-df1349bd44d4-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-57ff94957b-h49l4\" (UID: \"006c079d-a72d-40c3-8e75-df1349bd44d4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"
Sep 08 13:53:37.546655 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.546549    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/006c079d-a72d-40c3-8e75-df1349bd44d4-home\") pod \"tokenizer-template-args-test-kserve-57ff94957b-h49l4\" (UID: \"006c079d-a72d-40c3-8e75-df1349bd44d4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"
Sep 08 13:53:37.546655 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.546644    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/006c079d-a72d-40c3-8e75-df1349bd44d4-tls-certs\") pod \"tokenizer-template-args-test-kserve-57ff94957b-h49l4\" (UID: \"006c079d-a72d-40c3-8e75-df1349bd44d4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"
Sep 08 13:53:37.547100 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.547071    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/006c079d-a72d-40c3-8e75-df1349bd44d4-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-57ff94957b-h49l4\" (UID: \"006c079d-a72d-40c3-8e75-df1349bd44d4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"
Sep 08 13:53:37.548559 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.548531    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/006c079d-a72d-40c3-8e75-df1349bd44d4-dshm\") pod \"tokenizer-template-args-test-kserve-57ff94957b-h49l4\" (UID: \"006c079d-a72d-40c3-8e75-df1349bd44d4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"
Sep 08 13:53:37.548905 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.548884    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/006c079d-a72d-40c3-8e75-df1349bd44d4-tls-certs\") pod \"tokenizer-template-args-test-kserve-57ff94957b-h49l4\" (UID: \"006c079d-a72d-40c3-8e75-df1349bd44d4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"
Sep 08 13:53:37.554685 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.554666    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwcrv\" (UniqueName: \"kubernetes.io/projected/006c079d-a72d-40c3-8e75-df1349bd44d4-kube-api-access-fwcrv\") pod \"tokenizer-template-args-test-kserve-57ff94957b-h49l4\" (UID: \"006c079d-a72d-40c3-8e75-df1349bd44d4\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"
Sep 08 13:53:37.625611 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.625562    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb"]
Sep 08 13:53:37.630420 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.630390    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb"
Sep 08 13:53:37.638159 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.638123    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb"]
Sep 08 13:53:37.644919 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.644895    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"
Sep 08 13:53:37.748604 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.748556    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90a5d758-e41c-4133-936c-34cb500e2f04-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb\" (UID: \"90a5d758-e41c-4133-936c-34cb500e2f04\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb"
Sep 08 13:53:37.748765 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.748625    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90a5d758-e41c-4133-936c-34cb500e2f04-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb\" (UID: \"90a5d758-e41c-4133-936c-34cb500e2f04\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb"
Sep 08 13:53:37.748765 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.748703    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90a5d758-e41c-4133-936c-34cb500e2f04-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb\" (UID: \"90a5d758-e41c-4133-936c-34cb500e2f04\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb"
Sep 08 13:53:37.748765 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.748736    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a5d758-e41c-4133-936c-34cb500e2f04-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb\" (UID: \"90a5d758-e41c-4133-936c-34cb500e2f04\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb"
Sep 08 13:53:37.849398 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.849355    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90a5d758-e41c-4133-936c-34cb500e2f04-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb\" (UID: \"90a5d758-e41c-4133-936c-34cb500e2f04\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb"
Sep 08 13:53:37.849398 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.849405    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90a5d758-e41c-4133-936c-34cb500e2f04-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb\" (UID: \"90a5d758-e41c-4133-936c-34cb500e2f04\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb"
Sep 08 13:53:37.849645 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.849450    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90a5d758-e41c-4133-936c-34cb500e2f04-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb\" (UID: \"90a5d758-e41c-4133-936c-34cb500e2f04\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb"
Sep 08 13:53:37.849645 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.849468    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a5d758-e41c-4133-936c-34cb500e2f04-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb\" (UID: \"90a5d758-e41c-4133-936c-34cb500e2f04\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb"
Sep 08 13:53:37.849941 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.849916    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90a5d758-e41c-4133-936c-34cb500e2f04-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb\" (UID: \"90a5d758-e41c-4133-936c-34cb500e2f04\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb"
Sep 08 13:53:37.849941 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.849935    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a5d758-e41c-4133-936c-34cb500e2f04-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb\" (UID: \"90a5d758-e41c-4133-936c-34cb500e2f04\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb"
Sep 08 13:53:37.850256 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.850233    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90a5d758-e41c-4133-936c-34cb500e2f04-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb\" (UID: \"90a5d758-e41c-4133-936c-34cb500e2f04\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb"
Sep 08 13:53:37.851889 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.851872    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90a5d758-e41c-4133-936c-34cb500e2f04-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb\" (UID: \"90a5d758-e41c-4133-936c-34cb500e2f04\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb"
Sep 08 13:53:37.933751 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.933724    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"]
Sep 08 13:53:37.935708 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:53:37.935683    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod006c079d_a72d_40c3_8e75_df1349bd44d4.slice/crio-27945dced3a3f2401fe669b8b4a9ee20daa172688672884444414c6d70da5a2b WatchSource:0}: Error finding container 27945dced3a3f2401fe669b8b4a9ee20daa172688672884444414c6d70da5a2b: Status 404 returned error can't find the container with id 27945dced3a3f2401fe669b8b4a9ee20daa172688672884444414c6d70da5a2b
Sep 08 13:53:37.942474 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.942453    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb"
Sep 08 13:53:37.997973 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:37.997896    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4" event={"ID":"006c079d-a72d-40c3-8e75-df1349bd44d4","Type":"ContainerStarted","Data":"27945dced3a3f2401fe669b8b4a9ee20daa172688672884444414c6d70da5a2b"}
Sep 08 13:53:38.223106 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:38.223081    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb"]
Sep 08 13:53:38.226329 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:53:38.226300    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90a5d758_e41c_4133_936c_34cb500e2f04.slice/crio-1bbe752dc29de13181671644f7a236def45ef903a18f1c3af12984da7ea148fc WatchSource:0}: Error finding container 1bbe752dc29de13181671644f7a236def45ef903a18f1c3af12984da7ea148fc: Status 404 returned error can't find the container with id 1bbe752dc29de13181671644f7a236def45ef903a18f1c3af12984da7ea148fc
Sep 08 13:53:39.005049 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:39.005000    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb" event={"ID":"90a5d758-e41c-4133-936c-34cb500e2f04","Type":"ContainerStarted","Data":"be4d28db6c0c04bd5eb99de6cd84ae67d6807fa55184bb0605a81f6e28a33884"}
Sep 08 13:53:39.005507 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:39.005081    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb" event={"ID":"90a5d758-e41c-4133-936c-34cb500e2f04","Type":"ContainerStarted","Data":"1bbe752dc29de13181671644f7a236def45ef903a18f1c3af12984da7ea148fc"}
Sep 08 13:53:39.007313 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:39.007277    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4" event={"ID":"006c079d-a72d-40c3-8e75-df1349bd44d4","Type":"ContainerStarted","Data":"bb692f37c3e6eb9ea88d73710ac55e288512c583e0e2303cab6af1dd8700215f"}
Sep 08 13:53:41.021665 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:41.021632    2558 generic.go:356] "Generic (PLEG): container finished" podID="90a5d758-e41c-4133-936c-34cb500e2f04" containerID="be4d28db6c0c04bd5eb99de6cd84ae67d6807fa55184bb0605a81f6e28a33884" exitCode=0
Sep 08 13:53:41.022100 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:41.021710    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb" event={"ID":"90a5d758-e41c-4133-936c-34cb500e2f04","Type":"ContainerDied","Data":"be4d28db6c0c04bd5eb99de6cd84ae67d6807fa55184bb0605a81f6e28a33884"}
Sep 08 13:53:41.023555 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:41.023525    2558 generic.go:356] "Generic (PLEG): container finished" podID="006c079d-a72d-40c3-8e75-df1349bd44d4" containerID="bb692f37c3e6eb9ea88d73710ac55e288512c583e0e2303cab6af1dd8700215f" exitCode=0
Sep 08 13:53:41.023817 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:41.023622    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4" event={"ID":"006c079d-a72d-40c3-8e75-df1349bd44d4","Type":"ContainerDied","Data":"bb692f37c3e6eb9ea88d73710ac55e288512c583e0e2303cab6af1dd8700215f"}
Sep 08 13:53:42.033611 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:42.033556    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4" event={"ID":"006c079d-a72d-40c3-8e75-df1349bd44d4","Type":"ContainerStarted","Data":"bec303c83d785159e924b0a8cccf7c1227ca59e4507a54352d396a8ed7648ae2"}
Sep 08 13:53:42.035534 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:42.035510    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb" event={"ID":"90a5d758-e41c-4133-936c-34cb500e2f04","Type":"ContainerStarted","Data":"85cbde121a7e6cc4e1ff5459ee9b98104df401e07d9fd75346c8b2925fc3551b"}
Sep 08 13:53:42.035739 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:42.035720    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb"
Sep 08 13:53:42.060233 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:42.060189    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4" podStartSLOduration=2.4826067800000002 podStartE2EDuration="5.060174697s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.577567917s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:53:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:53:42.059726757 +0000 UTC m=+2285.511487687" watchObservedRunningTime="2026-09-08 13:53:42.060174697 +0000 UTC m=+2285.511935653"
Sep 08 13:53:42.078882 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:42.078831    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb" podStartSLOduration=2.527319287 podStartE2EDuration="5.078816151s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.551496864s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:53:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:53:42.077839387 +0000 UTC m=+2285.529600320" watchObservedRunningTime="2026-09-08 13:53:42.078816151 +0000 UTC m=+2285.530577103"
Sep 08 13:53:42.645530 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:42.645481    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"
Sep 08 13:53:47.645994 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:47.645952    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"
Sep 08 13:53:47.659406 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:47.659380    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"
Sep 08 13:53:48.080148 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:53:48.080067    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"
Sep 08 13:54:13.054156 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:13.054123    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb"
Sep 08 13:54:14.323300 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:14.323255    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb_90a5d758-e41c-4133-936c-34cb500e2f04/main/0.log"
Sep 08 13:54:14.560851 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:14.560817    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"]
Sep 08 13:54:14.570385 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:14.570355    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb"]
Sep 08 13:54:14.570661 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:14.570614    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb" podUID="90a5d758-e41c-4133-936c-34cb500e2f04" containerName="main" containerID="cri-o://85cbde121a7e6cc4e1ff5459ee9b98104df401e07d9fd75346c8b2925fc3551b" gracePeriod=30
Sep 08 13:54:14.571553 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:54:14.571520    2558 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 08 13:54:14.571553 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:54:14.571553 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:54:14.571553 ip-10-0-128-147 kubenswrapper[2558]:  > containerID="bec303c83d785159e924b0a8cccf7c1227ca59e4507a54352d396a8ed7648ae2" cmd=["/bin/sleep","15"]
Sep 08 13:54:14.571856 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:14.571579    2558 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4" message="" err=<
Sep 08 13:54:14.571856 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:54:14.571856 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:54:14.571856 ip-10-0-128-147 kubenswrapper[2558]:  >
Sep 08 13:54:14.571856 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:54:14.571618    2558 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 08 13:54:14.571856 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:54:14.571856 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:54:14.571856 ip-10-0-128-147 kubenswrapper[2558]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4" podUID="006c079d-a72d-40c3-8e75-df1349bd44d4" containerName="main" containerID="cri-o://bec303c83d785159e924b0a8cccf7c1227ca59e4507a54352d396a8ed7648ae2"
Sep 08 13:54:14.571856 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:14.571661    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4" podUID="006c079d-a72d-40c3-8e75-df1349bd44d4" containerName="main" containerID="cri-o://bec303c83d785159e924b0a8cccf7c1227ca59e4507a54352d396a8ed7648ae2" gracePeriod=60
Sep 08 13:54:14.928803 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:14.928731    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"
Sep 08 13:54:15.006537 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.006505    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/006c079d-a72d-40c3-8e75-df1349bd44d4-tmp-dir\") pod \"006c079d-a72d-40c3-8e75-df1349bd44d4\" (UID: \"006c079d-a72d-40c3-8e75-df1349bd44d4\") "
Sep 08 13:54:15.006738 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.006555    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/006c079d-a72d-40c3-8e75-df1349bd44d4-kserve-provision-location\") pod \"006c079d-a72d-40c3-8e75-df1349bd44d4\" (UID: \"006c079d-a72d-40c3-8e75-df1349bd44d4\") "
Sep 08 13:54:15.006738 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.006625    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/006c079d-a72d-40c3-8e75-df1349bd44d4-tls-certs\") pod \"006c079d-a72d-40c3-8e75-df1349bd44d4\" (UID: \"006c079d-a72d-40c3-8e75-df1349bd44d4\") "
Sep 08 13:54:15.006738 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.006649    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/006c079d-a72d-40c3-8e75-df1349bd44d4-model-cache\") pod \"006c079d-a72d-40c3-8e75-df1349bd44d4\" (UID: \"006c079d-a72d-40c3-8e75-df1349bd44d4\") "
Sep 08 13:54:15.006738 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.006710    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/006c079d-a72d-40c3-8e75-df1349bd44d4-dshm\") pod \"006c079d-a72d-40c3-8e75-df1349bd44d4\" (UID: \"006c079d-a72d-40c3-8e75-df1349bd44d4\") "
Sep 08 13:54:15.006984 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.006741    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/006c079d-a72d-40c3-8e75-df1349bd44d4-home\") pod \"006c079d-a72d-40c3-8e75-df1349bd44d4\" (UID: \"006c079d-a72d-40c3-8e75-df1349bd44d4\") "
Sep 08 13:54:15.006984 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.006784    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwcrv\" (UniqueName: \"kubernetes.io/projected/006c079d-a72d-40c3-8e75-df1349bd44d4-kube-api-access-fwcrv\") pod \"006c079d-a72d-40c3-8e75-df1349bd44d4\" (UID: \"006c079d-a72d-40c3-8e75-df1349bd44d4\") "
Sep 08 13:54:15.006984 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.006810    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/006c079d-a72d-40c3-8e75-df1349bd44d4-cabundle-cert\") pod \"006c079d-a72d-40c3-8e75-df1349bd44d4\" (UID: \"006c079d-a72d-40c3-8e75-df1349bd44d4\") "
Sep 08 13:54:15.006984 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.006843    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006c079d-a72d-40c3-8e75-df1349bd44d4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "006c079d-a72d-40c3-8e75-df1349bd44d4" (UID: "006c079d-a72d-40c3-8e75-df1349bd44d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:54:15.006984 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.006941    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006c079d-a72d-40c3-8e75-df1349bd44d4-model-cache" (OuterVolumeSpecName: "model-cache") pod "006c079d-a72d-40c3-8e75-df1349bd44d4" (UID: "006c079d-a72d-40c3-8e75-df1349bd44d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:54:15.007258 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.007059    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006c079d-a72d-40c3-8e75-df1349bd44d4-home" (OuterVolumeSpecName: "home") pod "006c079d-a72d-40c3-8e75-df1349bd44d4" (UID: "006c079d-a72d-40c3-8e75-df1349bd44d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:54:15.007430 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.007269    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/006c079d-a72d-40c3-8e75-df1349bd44d4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "006c079d-a72d-40c3-8e75-df1349bd44d4" (UID: "006c079d-a72d-40c3-8e75-df1349bd44d4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:54:15.007430 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.007197    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/006c079d-a72d-40c3-8e75-df1349bd44d4-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:54:15.007555 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.007442    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/006c079d-a72d-40c3-8e75-df1349bd44d4-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:54:15.008928 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.008892    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/006c079d-a72d-40c3-8e75-df1349bd44d4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "006c079d-a72d-40c3-8e75-df1349bd44d4" (UID: "006c079d-a72d-40c3-8e75-df1349bd44d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:54:15.009203 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.009177    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/006c079d-a72d-40c3-8e75-df1349bd44d4-kube-api-access-fwcrv" (OuterVolumeSpecName: "kube-api-access-fwcrv") pod "006c079d-a72d-40c3-8e75-df1349bd44d4" (UID: "006c079d-a72d-40c3-8e75-df1349bd44d4"). InnerVolumeSpecName "kube-api-access-fwcrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:54:15.009459 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.009430    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006c079d-a72d-40c3-8e75-df1349bd44d4-dshm" (OuterVolumeSpecName: "dshm") pod "006c079d-a72d-40c3-8e75-df1349bd44d4" (UID: "006c079d-a72d-40c3-8e75-df1349bd44d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:54:15.030680 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.030646    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006c079d-a72d-40c3-8e75-df1349bd44d4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "006c079d-a72d-40c3-8e75-df1349bd44d4" (UID: "006c079d-a72d-40c3-8e75-df1349bd44d4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:54:15.108333 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.108299    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwcrv\" (UniqueName: \"kubernetes.io/projected/006c079d-a72d-40c3-8e75-df1349bd44d4-kube-api-access-fwcrv\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:54:15.108333 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.108328    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/006c079d-a72d-40c3-8e75-df1349bd44d4-cabundle-cert\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:54:15.108333 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.108339    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/006c079d-a72d-40c3-8e75-df1349bd44d4-kserve-provision-location\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:54:15.108673 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.108348    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/006c079d-a72d-40c3-8e75-df1349bd44d4-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:54:15.108673 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.108356    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/006c079d-a72d-40c3-8e75-df1349bd44d4-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:54:15.108673 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.108368    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/006c079d-a72d-40c3-8e75-df1349bd44d4-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:54:15.136318 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.136288    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb"
Sep 08 13:54:15.208702 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.208588    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90a5d758-e41c-4133-936c-34cb500e2f04-kserve-provision-location\") pod \"90a5d758-e41c-4133-936c-34cb500e2f04\" (UID: \"90a5d758-e41c-4133-936c-34cb500e2f04\") "
Sep 08 13:54:15.208702 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.208653    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a5d758-e41c-4133-936c-34cb500e2f04-tmp-dir\") pod \"90a5d758-e41c-4133-936c-34cb500e2f04\" (UID: \"90a5d758-e41c-4133-936c-34cb500e2f04\") "
Sep 08 13:54:15.208935 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.208716    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90a5d758-e41c-4133-936c-34cb500e2f04-cabundle-cert\") pod \"90a5d758-e41c-4133-936c-34cb500e2f04\" (UID: \"90a5d758-e41c-4133-936c-34cb500e2f04\") "
Sep 08 13:54:15.208935 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.208806    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90a5d758-e41c-4133-936c-34cb500e2f04-tls-certs\") pod \"90a5d758-e41c-4133-936c-34cb500e2f04\" (UID: \"90a5d758-e41c-4133-936c-34cb500e2f04\") "
Sep 08 13:54:15.209063 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.209036    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a5d758-e41c-4133-936c-34cb500e2f04-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "90a5d758-e41c-4133-936c-34cb500e2f04" (UID: "90a5d758-e41c-4133-936c-34cb500e2f04"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:54:15.209118 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.209083    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90a5d758-e41c-4133-936c-34cb500e2f04-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "90a5d758-e41c-4133-936c-34cb500e2f04" (UID: "90a5d758-e41c-4133-936c-34cb500e2f04"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:54:15.209172 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.209148    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a5d758-e41c-4133-936c-34cb500e2f04-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:54:15.209997 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.209967    2558 generic.go:356] "Generic (PLEG): container finished" podID="90a5d758-e41c-4133-936c-34cb500e2f04" containerID="85cbde121a7e6cc4e1ff5459ee9b98104df401e07d9fd75346c8b2925fc3551b" exitCode=0
Sep 08 13:54:15.210127 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.210019    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb"
Sep 08 13:54:15.210127 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.210044    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb" event={"ID":"90a5d758-e41c-4133-936c-34cb500e2f04","Type":"ContainerDied","Data":"85cbde121a7e6cc4e1ff5459ee9b98104df401e07d9fd75346c8b2925fc3551b"}
Sep 08 13:54:15.210127 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.210105    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb" event={"ID":"90a5d758-e41c-4133-936c-34cb500e2f04","Type":"ContainerDied","Data":"1bbe752dc29de13181671644f7a236def45ef903a18f1c3af12984da7ea148fc"}
Sep 08 13:54:15.210311 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.210138    2558 scope.go:117] "RemoveContainer" containerID="85cbde121a7e6cc4e1ff5459ee9b98104df401e07d9fd75346c8b2925fc3551b"
Sep 08 13:54:15.211925 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.211816    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90a5d758-e41c-4133-936c-34cb500e2f04-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "90a5d758-e41c-4133-936c-34cb500e2f04" (UID: "90a5d758-e41c-4133-936c-34cb500e2f04"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:54:15.214030 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.213981    2558 generic.go:356] "Generic (PLEG): container finished" podID="006c079d-a72d-40c3-8e75-df1349bd44d4" containerID="bec303c83d785159e924b0a8cccf7c1227ca59e4507a54352d396a8ed7648ae2" exitCode=0
Sep 08 13:54:15.214030 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.214041    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"
Sep 08 13:54:15.214323 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.214107    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4" event={"ID":"006c079d-a72d-40c3-8e75-df1349bd44d4","Type":"ContainerDied","Data":"bec303c83d785159e924b0a8cccf7c1227ca59e4507a54352d396a8ed7648ae2"}
Sep 08 13:54:15.214323 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.214153    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4" event={"ID":"006c079d-a72d-40c3-8e75-df1349bd44d4","Type":"ContainerDied","Data":"27945dced3a3f2401fe669b8b4a9ee20daa172688672884444414c6d70da5a2b"}
Sep 08 13:54:15.222681 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.222662    2558 scope.go:117] "RemoveContainer" containerID="be4d28db6c0c04bd5eb99de6cd84ae67d6807fa55184bb0605a81f6e28a33884"
Sep 08 13:54:15.231171 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.231150    2558 scope.go:117] "RemoveContainer" containerID="85cbde121a7e6cc4e1ff5459ee9b98104df401e07d9fd75346c8b2925fc3551b"
Sep 08 13:54:15.231522 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:54:15.231495    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85cbde121a7e6cc4e1ff5459ee9b98104df401e07d9fd75346c8b2925fc3551b\": container with ID starting with 85cbde121a7e6cc4e1ff5459ee9b98104df401e07d9fd75346c8b2925fc3551b not found: ID does not exist" containerID="85cbde121a7e6cc4e1ff5459ee9b98104df401e07d9fd75346c8b2925fc3551b"
Sep 08 13:54:15.231632 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.231534    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85cbde121a7e6cc4e1ff5459ee9b98104df401e07d9fd75346c8b2925fc3551b"} err="failed to get container status \"85cbde121a7e6cc4e1ff5459ee9b98104df401e07d9fd75346c8b2925fc3551b\": rpc error: code = NotFound desc = could not find container \"85cbde121a7e6cc4e1ff5459ee9b98104df401e07d9fd75346c8b2925fc3551b\": container with ID starting with 85cbde121a7e6cc4e1ff5459ee9b98104df401e07d9fd75346c8b2925fc3551b not found: ID does not exist"
Sep 08 13:54:15.231632 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.231559    2558 scope.go:117] "RemoveContainer" containerID="be4d28db6c0c04bd5eb99de6cd84ae67d6807fa55184bb0605a81f6e28a33884"
Sep 08 13:54:15.231895 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:54:15.231874    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be4d28db6c0c04bd5eb99de6cd84ae67d6807fa55184bb0605a81f6e28a33884\": container with ID starting with be4d28db6c0c04bd5eb99de6cd84ae67d6807fa55184bb0605a81f6e28a33884 not found: ID does not exist" containerID="be4d28db6c0c04bd5eb99de6cd84ae67d6807fa55184bb0605a81f6e28a33884"
Sep 08 13:54:15.231943 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.231901    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be4d28db6c0c04bd5eb99de6cd84ae67d6807fa55184bb0605a81f6e28a33884"} err="failed to get container status \"be4d28db6c0c04bd5eb99de6cd84ae67d6807fa55184bb0605a81f6e28a33884\": rpc error: code = NotFound desc = could not find container \"be4d28db6c0c04bd5eb99de6cd84ae67d6807fa55184bb0605a81f6e28a33884\": container with ID starting with be4d28db6c0c04bd5eb99de6cd84ae67d6807fa55184bb0605a81f6e28a33884 not found: ID does not exist"
Sep 08 13:54:15.231943 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.231916    2558 scope.go:117] "RemoveContainer" containerID="bec303c83d785159e924b0a8cccf7c1227ca59e4507a54352d396a8ed7648ae2"
Sep 08 13:54:15.232408 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.232385    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"]
Sep 08 13:54:15.234905 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.234876    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a5d758-e41c-4133-936c-34cb500e2f04-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "90a5d758-e41c-4133-936c-34cb500e2f04" (UID: "90a5d758-e41c-4133-936c-34cb500e2f04"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:54:15.237550 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.237529    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-57ff94957b-h49l4"]
Sep 08 13:54:15.240753 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.240738    2558 scope.go:117] "RemoveContainer" containerID="bb692f37c3e6eb9ea88d73710ac55e288512c583e0e2303cab6af1dd8700215f"
Sep 08 13:54:15.248728 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.248711    2558 scope.go:117] "RemoveContainer" containerID="bec303c83d785159e924b0a8cccf7c1227ca59e4507a54352d396a8ed7648ae2"
Sep 08 13:54:15.249001 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:54:15.248984    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bec303c83d785159e924b0a8cccf7c1227ca59e4507a54352d396a8ed7648ae2\": container with ID starting with bec303c83d785159e924b0a8cccf7c1227ca59e4507a54352d396a8ed7648ae2 not found: ID does not exist" containerID="bec303c83d785159e924b0a8cccf7c1227ca59e4507a54352d396a8ed7648ae2"
Sep 08 13:54:15.249079 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.249012    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bec303c83d785159e924b0a8cccf7c1227ca59e4507a54352d396a8ed7648ae2"} err="failed to get container status \"bec303c83d785159e924b0a8cccf7c1227ca59e4507a54352d396a8ed7648ae2\": rpc error: code = NotFound desc = could not find container \"bec303c83d785159e924b0a8cccf7c1227ca59e4507a54352d396a8ed7648ae2\": container with ID starting with bec303c83d785159e924b0a8cccf7c1227ca59e4507a54352d396a8ed7648ae2 not found: ID does not exist"
Sep 08 13:54:15.249079 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.249033    2558 scope.go:117] "RemoveContainer" containerID="bb692f37c3e6eb9ea88d73710ac55e288512c583e0e2303cab6af1dd8700215f"
Sep 08 13:54:15.249305 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:54:15.249288    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb692f37c3e6eb9ea88d73710ac55e288512c583e0e2303cab6af1dd8700215f\": container with ID starting with bb692f37c3e6eb9ea88d73710ac55e288512c583e0e2303cab6af1dd8700215f not found: ID does not exist" containerID="bb692f37c3e6eb9ea88d73710ac55e288512c583e0e2303cab6af1dd8700215f"
Sep 08 13:54:15.249350 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.249311    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb692f37c3e6eb9ea88d73710ac55e288512c583e0e2303cab6af1dd8700215f"} err="failed to get container status \"bb692f37c3e6eb9ea88d73710ac55e288512c583e0e2303cab6af1dd8700215f\": rpc error: code = NotFound desc = could not find container \"bb692f37c3e6eb9ea88d73710ac55e288512c583e0e2303cab6af1dd8700215f\": container with ID starting with bb692f37c3e6eb9ea88d73710ac55e288512c583e0e2303cab6af1dd8700215f not found: ID does not exist"
Sep 08 13:54:15.310523 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.310489    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90a5d758-e41c-4133-936c-34cb500e2f04-kserve-provision-location\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:54:15.310523 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.310516    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/90a5d758-e41c-4133-936c-34cb500e2f04-cabundle-cert\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:54:15.310523 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.310527    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90a5d758-e41c-4133-936c-34cb500e2f04-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:54:15.533519 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.533477    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb"]
Sep 08 13:54:15.537293 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:15.537266    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-65c4987b6-s4kjb"]
Sep 08 13:54:17.053298 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:17.053264    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="006c079d-a72d-40c3-8e75-df1349bd44d4" path="/var/lib/kubelet/pods/006c079d-a72d-40c3-8e75-df1349bd44d4/volumes"
Sep 08 13:54:17.053754 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:17.053740    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90a5d758-e41c-4133-936c-34cb500e2f04" path="/var/lib/kubelet/pods/90a5d758-e41c-4133-936c-34cb500e2f04/volumes"
Sep 08 13:54:40.730941 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:40.730909    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl"]
Sep 08 13:54:41.128842 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:41.128803    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q"]
Sep 08 13:54:41.141119 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:54:41.141051    2558 secret.go:189] Couldn't get secret e2e-test-three-member-group-c33d828b/tri-v1-kserve-self-signed-certs: secret "tri-v1-kserve-self-signed-certs" not found
Sep 08 13:54:41.141276 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:54:41.141134    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a180d75f-2f98-4045-b642-7acd13bc94d4-tls-certs podName:a180d75f-2f98-4045-b642-7acd13bc94d4 nodeName:}" failed. No retries permitted until 2026-09-08 13:54:41.641113406 +0000 UTC m=+2345.092874321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a180d75f-2f98-4045-b642-7acd13bc94d4-tls-certs") pod "tri-v1-kserve-79f8f98bc5-wlk5q" (UID: "a180d75f-2f98-4045-b642-7acd13bc94d4") : secret "tri-v1-kserve-self-signed-certs" not found
Sep 08 13:54:41.645042 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:54:41.645003    2558 secret.go:189] Couldn't get secret e2e-test-three-member-group-c33d828b/tri-v1-kserve-self-signed-certs: secret "tri-v1-kserve-self-signed-certs" not found
Sep 08 13:54:41.645246 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:54:41.645079    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a180d75f-2f98-4045-b642-7acd13bc94d4-tls-certs podName:a180d75f-2f98-4045-b642-7acd13bc94d4 nodeName:}" failed. No retries permitted until 2026-09-08 13:54:42.645060312 +0000 UTC m=+2346.096821237 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a180d75f-2f98-4045-b642-7acd13bc94d4-tls-certs") pod "tri-v1-kserve-79f8f98bc5-wlk5q" (UID: "a180d75f-2f98-4045-b642-7acd13bc94d4") : secret "tri-v1-kserve-self-signed-certs" not found
Sep 08 13:54:41.927985 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:41.927902    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm"]
Sep 08 13:54:42.656358 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:54:42.656316    2558 secret.go:189] Couldn't get secret e2e-test-three-member-group-c33d828b/tri-v1-kserve-self-signed-certs: secret "tri-v1-kserve-self-signed-certs" not found
Sep 08 13:54:42.656580 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:54:42.656421    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a180d75f-2f98-4045-b642-7acd13bc94d4-tls-certs podName:a180d75f-2f98-4045-b642-7acd13bc94d4 nodeName:}" failed. No retries permitted until 2026-09-08 13:54:44.656403489 +0000 UTC m=+2348.108164415 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a180d75f-2f98-4045-b642-7acd13bc94d4-tls-certs") pod "tri-v1-kserve-79f8f98bc5-wlk5q" (UID: "a180d75f-2f98-4045-b642-7acd13bc94d4") : secret "tri-v1-kserve-self-signed-certs" not found
Sep 08 13:54:44.676053 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:54:44.676001    2558 secret.go:189] Couldn't get secret e2e-test-three-member-group-c33d828b/tri-v1-kserve-self-signed-certs: secret "tri-v1-kserve-self-signed-certs" not found
Sep 08 13:54:44.676468 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:54:44.676086    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a180d75f-2f98-4045-b642-7acd13bc94d4-tls-certs podName:a180d75f-2f98-4045-b642-7acd13bc94d4 nodeName:}" failed. No retries permitted until 2026-09-08 13:54:48.67607162 +0000 UTC m=+2352.127832529 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a180d75f-2f98-4045-b642-7acd13bc94d4-tls-certs") pod "tri-v1-kserve-79f8f98bc5-wlk5q" (UID: "a180d75f-2f98-4045-b642-7acd13bc94d4") : secret "tri-v1-kserve-self-signed-certs" not found
Sep 08 13:54:45.413363 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.413327    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9"]
Sep 08 13:54:45.413724 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.413710    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="006c079d-a72d-40c3-8e75-df1349bd44d4" containerName="main"
Sep 08 13:54:45.413780 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.413727    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="006c079d-a72d-40c3-8e75-df1349bd44d4" containerName="main"
Sep 08 13:54:45.413780 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.413742    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90a5d758-e41c-4133-936c-34cb500e2f04" containerName="storage-initializer"
Sep 08 13:54:45.413780 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.413747    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="90a5d758-e41c-4133-936c-34cb500e2f04" containerName="storage-initializer"
Sep 08 13:54:45.413780 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.413757    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90a5d758-e41c-4133-936c-34cb500e2f04" containerName="main"
Sep 08 13:54:45.413780 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.413762    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="90a5d758-e41c-4133-936c-34cb500e2f04" containerName="main"
Sep 08 13:54:45.413780 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.413778    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="006c079d-a72d-40c3-8e75-df1349bd44d4" containerName="storage-initializer"
Sep 08 13:54:45.413967 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.413783    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="006c079d-a72d-40c3-8e75-df1349bd44d4" containerName="storage-initializer"
Sep 08 13:54:45.413967 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.413837    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="90a5d758-e41c-4133-936c-34cb500e2f04" containerName="main"
Sep 08 13:54:45.413967 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.413847    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="006c079d-a72d-40c3-8e75-df1349bd44d4" containerName="main"
Sep 08 13:54:45.416858 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.416841    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9"
Sep 08 13:54:45.419333 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.419303    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\""
Sep 08 13:54:45.420153 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.420133    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\""
Sep 08 13:54:45.420250 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.420153    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v1-kserve-self-signed-certs\""
Sep 08 13:54:45.420250 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.420170    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-7mq9x\""
Sep 08 13:54:45.427766 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.427739    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9"]
Sep 08 13:54:45.484565 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.484529    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdltq\" (UniqueName: \"kubernetes.io/projected/2e682304-c831-4b80-b597-597383c3a776-kube-api-access-vdltq\") pod \"late-v1-kserve-56cbc8456c-pq6j9\" (UID: \"2e682304-c831-4b80-b597-597383c3a776\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9"
Sep 08 13:54:45.484565 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.484569    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e682304-c831-4b80-b597-597383c3a776-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-pq6j9\" (UID: \"2e682304-c831-4b80-b597-597383c3a776\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9"
Sep 08 13:54:45.484824 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.484585    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e682304-c831-4b80-b597-597383c3a776-tls-certs\") pod \"late-v1-kserve-56cbc8456c-pq6j9\" (UID: \"2e682304-c831-4b80-b597-597383c3a776\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9"
Sep 08 13:54:45.484824 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.484690    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e682304-c831-4b80-b597-597383c3a776-model-cache\") pod \"late-v1-kserve-56cbc8456c-pq6j9\" (UID: \"2e682304-c831-4b80-b597-597383c3a776\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9"
Sep 08 13:54:45.484824 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.484731    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e682304-c831-4b80-b597-597383c3a776-home\") pod \"late-v1-kserve-56cbc8456c-pq6j9\" (UID: \"2e682304-c831-4b80-b597-597383c3a776\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9"
Sep 08 13:54:45.484824 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.484751    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e682304-c831-4b80-b597-597383c3a776-dshm\") pod \"late-v1-kserve-56cbc8456c-pq6j9\" (UID: \"2e682304-c831-4b80-b597-597383c3a776\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9"
Sep 08 13:54:45.585652 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.585588    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e682304-c831-4b80-b597-597383c3a776-home\") pod \"late-v1-kserve-56cbc8456c-pq6j9\" (UID: \"2e682304-c831-4b80-b597-597383c3a776\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9"
Sep 08 13:54:45.585652 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.585655    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e682304-c831-4b80-b597-597383c3a776-dshm\") pod \"late-v1-kserve-56cbc8456c-pq6j9\" (UID: \"2e682304-c831-4b80-b597-597383c3a776\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9"
Sep 08 13:54:45.585888 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.585708    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdltq\" (UniqueName: \"kubernetes.io/projected/2e682304-c831-4b80-b597-597383c3a776-kube-api-access-vdltq\") pod \"late-v1-kserve-56cbc8456c-pq6j9\" (UID: \"2e682304-c831-4b80-b597-597383c3a776\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9"
Sep 08 13:54:45.585888 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.585730    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e682304-c831-4b80-b597-597383c3a776-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-pq6j9\" (UID: \"2e682304-c831-4b80-b597-597383c3a776\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9"
Sep 08 13:54:45.585888 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.585745    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e682304-c831-4b80-b597-597383c3a776-tls-certs\") pod \"late-v1-kserve-56cbc8456c-pq6j9\" (UID: \"2e682304-c831-4b80-b597-597383c3a776\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9"
Sep 08 13:54:45.585888 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.585806    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e682304-c831-4b80-b597-597383c3a776-model-cache\") pod \"late-v1-kserve-56cbc8456c-pq6j9\" (UID: \"2e682304-c831-4b80-b597-597383c3a776\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9"
Sep 08 13:54:45.586100 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.586057    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e682304-c831-4b80-b597-597383c3a776-home\") pod \"late-v1-kserve-56cbc8456c-pq6j9\" (UID: \"2e682304-c831-4b80-b597-597383c3a776\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9"
Sep 08 13:54:45.586160 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.586135    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e682304-c831-4b80-b597-597383c3a776-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-pq6j9\" (UID: \"2e682304-c831-4b80-b597-597383c3a776\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9"
Sep 08 13:54:45.586217 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.586156    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e682304-c831-4b80-b597-597383c3a776-model-cache\") pod \"late-v1-kserve-56cbc8456c-pq6j9\" (UID: \"2e682304-c831-4b80-b597-597383c3a776\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9"
Sep 08 13:54:45.587915 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.587892    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e682304-c831-4b80-b597-597383c3a776-dshm\") pod \"late-v1-kserve-56cbc8456c-pq6j9\" (UID: \"2e682304-c831-4b80-b597-597383c3a776\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9"
Sep 08 13:54:45.588340 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.588321    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e682304-c831-4b80-b597-597383c3a776-tls-certs\") pod \"late-v1-kserve-56cbc8456c-pq6j9\" (UID: \"2e682304-c831-4b80-b597-597383c3a776\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9"
Sep 08 13:54:45.595064 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.595042    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdltq\" (UniqueName: \"kubernetes.io/projected/2e682304-c831-4b80-b597-597383c3a776-kube-api-access-vdltq\") pod \"late-v1-kserve-56cbc8456c-pq6j9\" (UID: \"2e682304-c831-4b80-b597-597383c3a776\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9"
Sep 08 13:54:45.727947 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:45.727849    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9"
Sep 08 13:54:46.004128 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:46.003978    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9"]
Sep 08 13:54:46.006638 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:54:46.006584    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e682304_c831_4b80_b597_597383c3a776.slice/crio-268e9621b8a15c74311142c8435e1e1cefded08d433288ffd74796a487907e15 WatchSource:0}: Error finding container 268e9621b8a15c74311142c8435e1e1cefded08d433288ffd74796a487907e15: Status 404 returned error can't find the container with id 268e9621b8a15c74311142c8435e1e1cefded08d433288ffd74796a487907e15
Sep 08 13:54:46.371728 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:46.371693    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9" event={"ID":"2e682304-c831-4b80-b597-597383c3a776","Type":"ContainerStarted","Data":"3e3075dc8714c31fa80abd975d7b72fea8beb2c087037d81b18b62b6b8c6bc09"}
Sep 08 13:54:46.371913 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:46.371758    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9" event={"ID":"2e682304-c831-4b80-b597-597383c3a776","Type":"ContainerStarted","Data":"268e9621b8a15c74311142c8435e1e1cefded08d433288ffd74796a487907e15"}
Sep 08 13:54:46.390773 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:46.390720    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9" podStartSLOduration=1.390702765 podStartE2EDuration="1.390702765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:54:46.389274857 +0000 UTC m=+2349.841035799" watchObservedRunningTime="2026-09-08 13:54:46.390702765 +0000 UTC m=+2349.842463697"
Sep 08 13:54:46.728038 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:46.727955    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9"
Sep 08 13:54:48.716819 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:54:48.716786    2558 secret.go:189] Couldn't get secret e2e-test-three-member-group-c33d828b/tri-v1-kserve-self-signed-certs: secret "tri-v1-kserve-self-signed-certs" not found
Sep 08 13:54:48.717232 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:54:48.716856    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a180d75f-2f98-4045-b642-7acd13bc94d4-tls-certs podName:a180d75f-2f98-4045-b642-7acd13bc94d4 nodeName:}" failed. No retries permitted until 2026-09-08 13:54:56.716841484 +0000 UTC m=+2360.168602393 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a180d75f-2f98-4045-b642-7acd13bc94d4-tls-certs") pod "tri-v1-kserve-79f8f98bc5-wlk5q" (UID: "a180d75f-2f98-4045-b642-7acd13bc94d4") : secret "tri-v1-kserve-self-signed-certs" not found
Sep 08 13:54:55.729036 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:55.728999    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9"
Sep 08 13:54:55.741824 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:55.741800    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9"
Sep 08 13:54:55.750878 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:55.750848    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl" podUID="92d05aef-6bc0-47ae-8abc-ab11a6416c2d" containerName="main" containerID="cri-o://47ba65954c50a24005a910f9cdd5236057591369b959a24a010217c5f7f20ca8" gracePeriod=45
Sep 08 13:54:56.006296 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.006266    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl"
Sep 08 13:54:56.087187 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.087158    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-tmp-dir\") pod \"92d05aef-6bc0-47ae-8abc-ab11a6416c2d\" (UID: \"92d05aef-6bc0-47ae-8abc-ab11a6416c2d\") "
Sep 08 13:54:56.087350 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.087207    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-home\") pod \"92d05aef-6bc0-47ae-8abc-ab11a6416c2d\" (UID: \"92d05aef-6bc0-47ae-8abc-ab11a6416c2d\") "
Sep 08 13:54:56.087350 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.087241    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-tls-certs\") pod \"92d05aef-6bc0-47ae-8abc-ab11a6416c2d\" (UID: \"92d05aef-6bc0-47ae-8abc-ab11a6416c2d\") "
Sep 08 13:54:56.087350 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.087268    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-model-cache\") pod \"92d05aef-6bc0-47ae-8abc-ab11a6416c2d\" (UID: \"92d05aef-6bc0-47ae-8abc-ab11a6416c2d\") "
Sep 08 13:54:56.087350 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.087338    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggzdp\" (UniqueName: \"kubernetes.io/projected/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-kube-api-access-ggzdp\") pod \"92d05aef-6bc0-47ae-8abc-ab11a6416c2d\" (UID: \"92d05aef-6bc0-47ae-8abc-ab11a6416c2d\") "
Sep 08 13:54:56.087557 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.087366    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-dshm\") pod \"92d05aef-6bc0-47ae-8abc-ab11a6416c2d\" (UID: \"92d05aef-6bc0-47ae-8abc-ab11a6416c2d\") "
Sep 08 13:54:56.087557 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.087427    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "92d05aef-6bc0-47ae-8abc-ab11a6416c2d" (UID: "92d05aef-6bc0-47ae-8abc-ab11a6416c2d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:54:56.087716 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.087531    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-home" (OuterVolumeSpecName: "home") pod "92d05aef-6bc0-47ae-8abc-ab11a6416c2d" (UID: "92d05aef-6bc0-47ae-8abc-ab11a6416c2d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:54:56.087716 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.087629    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-model-cache" (OuterVolumeSpecName: "model-cache") pod "92d05aef-6bc0-47ae-8abc-ab11a6416c2d" (UID: "92d05aef-6bc0-47ae-8abc-ab11a6416c2d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:54:56.087716 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.087702    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:54:56.087716 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.087718    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:54:56.087880 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.087726    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:54:56.089761 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.089733    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "92d05aef-6bc0-47ae-8abc-ab11a6416c2d" (UID: "92d05aef-6bc0-47ae-8abc-ab11a6416c2d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:54:56.089761 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.089750    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-kube-api-access-ggzdp" (OuterVolumeSpecName: "kube-api-access-ggzdp") pod "92d05aef-6bc0-47ae-8abc-ab11a6416c2d" (UID: "92d05aef-6bc0-47ae-8abc-ab11a6416c2d"). InnerVolumeSpecName "kube-api-access-ggzdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:54:56.089899 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.089777    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-dshm" (OuterVolumeSpecName: "dshm") pod "92d05aef-6bc0-47ae-8abc-ab11a6416c2d" (UID: "92d05aef-6bc0-47ae-8abc-ab11a6416c2d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:54:56.188898 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.188858    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:54:56.188898 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.188890    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggzdp\" (UniqueName: \"kubernetes.io/projected/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-kube-api-access-ggzdp\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:54:56.188898 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.188900    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92d05aef-6bc0-47ae-8abc-ab11a6416c2d-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:54:56.364561 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.364519    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q" podUID="a180d75f-2f98-4045-b642-7acd13bc94d4" containerName="main" containerID="cri-o://7d9ef70df098973338be598021f318afa563877258454661ca0b65f1beea631e" gracePeriod=45
Sep 08 13:54:56.418526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.418493    2558 generic.go:356] "Generic (PLEG): container finished" podID="92d05aef-6bc0-47ae-8abc-ab11a6416c2d" containerID="47ba65954c50a24005a910f9cdd5236057591369b959a24a010217c5f7f20ca8" exitCode=0
Sep 08 13:54:56.418660 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.418546    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl"
Sep 08 13:54:56.418660 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.418584    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl" event={"ID":"92d05aef-6bc0-47ae-8abc-ab11a6416c2d","Type":"ContainerDied","Data":"47ba65954c50a24005a910f9cdd5236057591369b959a24a010217c5f7f20ca8"}
Sep 08 13:54:56.418747 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.418663    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl" event={"ID":"92d05aef-6bc0-47ae-8abc-ab11a6416c2d","Type":"ContainerDied","Data":"ffd36c93000a30321ce42c711055281c7eb3a4d23ec02e6dfd20b37fd922f40f"}
Sep 08 13:54:56.418747 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.418687    2558 scope.go:117] "RemoveContainer" containerID="47ba65954c50a24005a910f9cdd5236057591369b959a24a010217c5f7f20ca8"
Sep 08 13:54:56.430497 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.430477    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9"
Sep 08 13:54:56.445968 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.445942    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl"]
Sep 08 13:54:56.450536 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.450516    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-fgzjl"]
Sep 08 13:54:56.475545 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.475529    2558 scope.go:117] "RemoveContainer" containerID="47ba65954c50a24005a910f9cdd5236057591369b959a24a010217c5f7f20ca8"
Sep 08 13:54:56.475942 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:54:56.475919    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47ba65954c50a24005a910f9cdd5236057591369b959a24a010217c5f7f20ca8\": container with ID starting with 47ba65954c50a24005a910f9cdd5236057591369b959a24a010217c5f7f20ca8 not found: ID does not exist" containerID="47ba65954c50a24005a910f9cdd5236057591369b959a24a010217c5f7f20ca8"
Sep 08 13:54:56.476005 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.475957    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47ba65954c50a24005a910f9cdd5236057591369b959a24a010217c5f7f20ca8"} err="failed to get container status \"47ba65954c50a24005a910f9cdd5236057591369b959a24a010217c5f7f20ca8\": rpc error: code = NotFound desc = could not find container \"47ba65954c50a24005a910f9cdd5236057591369b959a24a010217c5f7f20ca8\": container with ID starting with 47ba65954c50a24005a910f9cdd5236057591369b959a24a010217c5f7f20ca8 not found: ID does not exist"
Sep 08 13:54:56.598262 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.598231    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q"
Sep 08 13:54:56.692539 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.692464    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a180d75f-2f98-4045-b642-7acd13bc94d4-tls-certs\") pod \"a180d75f-2f98-4045-b642-7acd13bc94d4\" (UID: \"a180d75f-2f98-4045-b642-7acd13bc94d4\") "
Sep 08 13:54:56.692539 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.692496    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcmwt\" (UniqueName: \"kubernetes.io/projected/a180d75f-2f98-4045-b642-7acd13bc94d4-kube-api-access-vcmwt\") pod \"a180d75f-2f98-4045-b642-7acd13bc94d4\" (UID: \"a180d75f-2f98-4045-b642-7acd13bc94d4\") "
Sep 08 13:54:56.692784 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.692556    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a180d75f-2f98-4045-b642-7acd13bc94d4-tmp-dir\") pod \"a180d75f-2f98-4045-b642-7acd13bc94d4\" (UID: \"a180d75f-2f98-4045-b642-7acd13bc94d4\") "
Sep 08 13:54:56.692784 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.692578    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a180d75f-2f98-4045-b642-7acd13bc94d4-model-cache\") pod \"a180d75f-2f98-4045-b642-7acd13bc94d4\" (UID: \"a180d75f-2f98-4045-b642-7acd13bc94d4\") "
Sep 08 13:54:56.692784 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.692624    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a180d75f-2f98-4045-b642-7acd13bc94d4-dshm\") pod \"a180d75f-2f98-4045-b642-7acd13bc94d4\" (UID: \"a180d75f-2f98-4045-b642-7acd13bc94d4\") "
Sep 08 13:54:56.692784 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.692692    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a180d75f-2f98-4045-b642-7acd13bc94d4-home\") pod \"a180d75f-2f98-4045-b642-7acd13bc94d4\" (UID: \"a180d75f-2f98-4045-b642-7acd13bc94d4\") "
Sep 08 13:54:56.692996 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.692870    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a180d75f-2f98-4045-b642-7acd13bc94d4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a180d75f-2f98-4045-b642-7acd13bc94d4" (UID: "a180d75f-2f98-4045-b642-7acd13bc94d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:54:56.692996 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.692889    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a180d75f-2f98-4045-b642-7acd13bc94d4-model-cache" (OuterVolumeSpecName: "model-cache") pod "a180d75f-2f98-4045-b642-7acd13bc94d4" (UID: "a180d75f-2f98-4045-b642-7acd13bc94d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:54:56.692996 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.692933    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a180d75f-2f98-4045-b642-7acd13bc94d4-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:54:56.693105 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.693015    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a180d75f-2f98-4045-b642-7acd13bc94d4-home" (OuterVolumeSpecName: "home") pod "a180d75f-2f98-4045-b642-7acd13bc94d4" (UID: "a180d75f-2f98-4045-b642-7acd13bc94d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:54:56.694764 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.694739    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a180d75f-2f98-4045-b642-7acd13bc94d4-dshm" (OuterVolumeSpecName: "dshm") pod "a180d75f-2f98-4045-b642-7acd13bc94d4" (UID: "a180d75f-2f98-4045-b642-7acd13bc94d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:54:56.694826 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.694779    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a180d75f-2f98-4045-b642-7acd13bc94d4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a180d75f-2f98-4045-b642-7acd13bc94d4" (UID: "a180d75f-2f98-4045-b642-7acd13bc94d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:54:56.694826 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.694785    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a180d75f-2f98-4045-b642-7acd13bc94d4-kube-api-access-vcmwt" (OuterVolumeSpecName: "kube-api-access-vcmwt") pod "a180d75f-2f98-4045-b642-7acd13bc94d4" (UID: "a180d75f-2f98-4045-b642-7acd13bc94d4"). InnerVolumeSpecName "kube-api-access-vcmwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:54:56.794203 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.794174    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a180d75f-2f98-4045-b642-7acd13bc94d4-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:54:56.794203 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.794200    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a180d75f-2f98-4045-b642-7acd13bc94d4-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:54:56.794568 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.794208    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a180d75f-2f98-4045-b642-7acd13bc94d4-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:54:56.794568 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.794217    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a180d75f-2f98-4045-b642-7acd13bc94d4-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:54:56.794568 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.794226    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcmwt\" (UniqueName: \"kubernetes.io/projected/a180d75f-2f98-4045-b642-7acd13bc94d4-kube-api-access-vcmwt\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:54:56.948566 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:56.948471    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm" podUID="9864420c-5be6-442d-9381-79db08992db3" containerName="main" containerID="cri-o://2ed645a0d66f87a2ac7a3c59241e73cca725abde9d3de63b4c6c8184ff3a4e5a" gracePeriod=45
Sep 08 13:54:57.054398 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.054361    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92d05aef-6bc0-47ae-8abc-ab11a6416c2d" path="/var/lib/kubelet/pods/92d05aef-6bc0-47ae-8abc-ab11a6416c2d/volumes"
Sep 08 13:54:57.210402 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.210339    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm"
Sep 08 13:54:57.298844 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.298807    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9864420c-5be6-442d-9381-79db08992db3-tls-certs\") pod \"9864420c-5be6-442d-9381-79db08992db3\" (UID: \"9864420c-5be6-442d-9381-79db08992db3\") "
Sep 08 13:54:57.298844 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.298846    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9864420c-5be6-442d-9381-79db08992db3-model-cache\") pod \"9864420c-5be6-442d-9381-79db08992db3\" (UID: \"9864420c-5be6-442d-9381-79db08992db3\") "
Sep 08 13:54:57.299107 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.298868    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9864420c-5be6-442d-9381-79db08992db3-dshm\") pod \"9864420c-5be6-442d-9381-79db08992db3\" (UID: \"9864420c-5be6-442d-9381-79db08992db3\") "
Sep 08 13:54:57.299107 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.298912    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlzw5\" (UniqueName: \"kubernetes.io/projected/9864420c-5be6-442d-9381-79db08992db3-kube-api-access-qlzw5\") pod \"9864420c-5be6-442d-9381-79db08992db3\" (UID: \"9864420c-5be6-442d-9381-79db08992db3\") "
Sep 08 13:54:57.299107 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.298948    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9864420c-5be6-442d-9381-79db08992db3-tmp-dir\") pod \"9864420c-5be6-442d-9381-79db08992db3\" (UID: \"9864420c-5be6-442d-9381-79db08992db3\") "
Sep 08 13:54:57.299107 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.298979    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9864420c-5be6-442d-9381-79db08992db3-home\") pod \"9864420c-5be6-442d-9381-79db08992db3\" (UID: \"9864420c-5be6-442d-9381-79db08992db3\") "
Sep 08 13:54:57.299316 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.299165    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9864420c-5be6-442d-9381-79db08992db3-model-cache" (OuterVolumeSpecName: "model-cache") pod "9864420c-5be6-442d-9381-79db08992db3" (UID: "9864420c-5be6-442d-9381-79db08992db3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:54:57.299316 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.299248    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9864420c-5be6-442d-9381-79db08992db3-home" (OuterVolumeSpecName: "home") pod "9864420c-5be6-442d-9381-79db08992db3" (UID: "9864420c-5be6-442d-9381-79db08992db3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:54:57.299429 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.299379    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9864420c-5be6-442d-9381-79db08992db3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9864420c-5be6-442d-9381-79db08992db3" (UID: "9864420c-5be6-442d-9381-79db08992db3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:54:57.301197 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.301159    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9864420c-5be6-442d-9381-79db08992db3-dshm" (OuterVolumeSpecName: "dshm") pod "9864420c-5be6-442d-9381-79db08992db3" (UID: "9864420c-5be6-442d-9381-79db08992db3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:54:57.301197 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.301173    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9864420c-5be6-442d-9381-79db08992db3-kube-api-access-qlzw5" (OuterVolumeSpecName: "kube-api-access-qlzw5") pod "9864420c-5be6-442d-9381-79db08992db3" (UID: "9864420c-5be6-442d-9381-79db08992db3"). InnerVolumeSpecName "kube-api-access-qlzw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:54:57.301393 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.301257    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9864420c-5be6-442d-9381-79db08992db3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9864420c-5be6-442d-9381-79db08992db3" (UID: "9864420c-5be6-442d-9381-79db08992db3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:54:57.399971 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.399937    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9864420c-5be6-442d-9381-79db08992db3-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:54:57.399971 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.399965    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlzw5\" (UniqueName: \"kubernetes.io/projected/9864420c-5be6-442d-9381-79db08992db3-kube-api-access-qlzw5\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:54:57.399971 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.399978    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9864420c-5be6-442d-9381-79db08992db3-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:54:57.400207 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.399988    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9864420c-5be6-442d-9381-79db08992db3-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:54:57.400207 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.399995    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9864420c-5be6-442d-9381-79db08992db3-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:54:57.400207 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.400003    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9864420c-5be6-442d-9381-79db08992db3-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:54:57.427019 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.426985    2558 generic.go:356] "Generic (PLEG): container finished" podID="9864420c-5be6-442d-9381-79db08992db3" containerID="2ed645a0d66f87a2ac7a3c59241e73cca725abde9d3de63b4c6c8184ff3a4e5a" exitCode=0
Sep 08 13:54:57.427157 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.427043    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm"
Sep 08 13:54:57.427157 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.427076    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm" event={"ID":"9864420c-5be6-442d-9381-79db08992db3","Type":"ContainerDied","Data":"2ed645a0d66f87a2ac7a3c59241e73cca725abde9d3de63b4c6c8184ff3a4e5a"}
Sep 08 13:54:57.427302 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.427169    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm" event={"ID":"9864420c-5be6-442d-9381-79db08992db3","Type":"ContainerDied","Data":"1d085cf9c3c3e6bb34705468820329ee9023e65cb669830def33609747265bdc"}
Sep 08 13:54:57.427302 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.427198    2558 scope.go:117] "RemoveContainer" containerID="2ed645a0d66f87a2ac7a3c59241e73cca725abde9d3de63b4c6c8184ff3a4e5a"
Sep 08 13:54:57.428496 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.428477    2558 generic.go:356] "Generic (PLEG): container finished" podID="a180d75f-2f98-4045-b642-7acd13bc94d4" containerID="7d9ef70df098973338be598021f318afa563877258454661ca0b65f1beea631e" exitCode=0
Sep 08 13:54:57.428574 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.428523    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q"
Sep 08 13:54:57.428574 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.428557    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q" event={"ID":"a180d75f-2f98-4045-b642-7acd13bc94d4","Type":"ContainerDied","Data":"7d9ef70df098973338be598021f318afa563877258454661ca0b65f1beea631e"}
Sep 08 13:54:57.428735 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.428628    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q" event={"ID":"a180d75f-2f98-4045-b642-7acd13bc94d4","Type":"ContainerDied","Data":"2e83c9cb5203dd1bf8ca5a6a00864580b50c34fa7065f929edd8e3e3edb0decc"}
Sep 08 13:54:57.436374 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.436356    2558 scope.go:117] "RemoveContainer" containerID="2ed645a0d66f87a2ac7a3c59241e73cca725abde9d3de63b4c6c8184ff3a4e5a"
Sep 08 13:54:57.436697 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:54:57.436681    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ed645a0d66f87a2ac7a3c59241e73cca725abde9d3de63b4c6c8184ff3a4e5a\": container with ID starting with 2ed645a0d66f87a2ac7a3c59241e73cca725abde9d3de63b4c6c8184ff3a4e5a not found: ID does not exist" containerID="2ed645a0d66f87a2ac7a3c59241e73cca725abde9d3de63b4c6c8184ff3a4e5a"
Sep 08 13:54:57.436738 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.436704    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ed645a0d66f87a2ac7a3c59241e73cca725abde9d3de63b4c6c8184ff3a4e5a"} err="failed to get container status \"2ed645a0d66f87a2ac7a3c59241e73cca725abde9d3de63b4c6c8184ff3a4e5a\": rpc error: code = NotFound desc = could not find container \"2ed645a0d66f87a2ac7a3c59241e73cca725abde9d3de63b4c6c8184ff3a4e5a\": container with ID starting with 2ed645a0d66f87a2ac7a3c59241e73cca725abde9d3de63b4c6c8184ff3a4e5a not found: ID does not exist"
Sep 08 13:54:57.436738 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.436721    2558 scope.go:117] "RemoveContainer" containerID="7d9ef70df098973338be598021f318afa563877258454661ca0b65f1beea631e"
Sep 08 13:54:57.444615 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.444569    2558 scope.go:117] "RemoveContainer" containerID="7d9ef70df098973338be598021f318afa563877258454661ca0b65f1beea631e"
Sep 08 13:54:57.444914 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:54:57.444895    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d9ef70df098973338be598021f318afa563877258454661ca0b65f1beea631e\": container with ID starting with 7d9ef70df098973338be598021f318afa563877258454661ca0b65f1beea631e not found: ID does not exist" containerID="7d9ef70df098973338be598021f318afa563877258454661ca0b65f1beea631e"
Sep 08 13:54:57.444996 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.444920    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d9ef70df098973338be598021f318afa563877258454661ca0b65f1beea631e"} err="failed to get container status \"7d9ef70df098973338be598021f318afa563877258454661ca0b65f1beea631e\": rpc error: code = NotFound desc = could not find container \"7d9ef70df098973338be598021f318afa563877258454661ca0b65f1beea631e\": container with ID starting with 7d9ef70df098973338be598021f318afa563877258454661ca0b65f1beea631e not found: ID does not exist"
Sep 08 13:54:57.446966 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.446943    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q"]
Sep 08 13:54:57.452229 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.452208    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wlk5q"]
Sep 08 13:54:57.465347 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.465280    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm"]
Sep 08 13:54:57.467095 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:57.467075    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-pctqm"]
Sep 08 13:54:59.053501 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:59.053469    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9864420c-5be6-442d-9381-79db08992db3" path="/var/lib/kubelet/pods/9864420c-5be6-442d-9381-79db08992db3/volumes"
Sep 08 13:54:59.053889 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:54:59.053872    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a180d75f-2f98-4045-b642-7acd13bc94d4" path="/var/lib/kubelet/pods/a180d75f-2f98-4045-b642-7acd13bc94d4/volumes"
Sep 08 13:55:16.511698 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.511657    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs"]
Sep 08 13:55:16.512236 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.512216    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9864420c-5be6-442d-9381-79db08992db3" containerName="main"
Sep 08 13:55:16.512290 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.512240    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9864420c-5be6-442d-9381-79db08992db3" containerName="main"
Sep 08 13:55:16.512290 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.512282    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a180d75f-2f98-4045-b642-7acd13bc94d4" containerName="main"
Sep 08 13:55:16.512351 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.512293    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a180d75f-2f98-4045-b642-7acd13bc94d4" containerName="main"
Sep 08 13:55:16.512351 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.512310    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92d05aef-6bc0-47ae-8abc-ab11a6416c2d" containerName="main"
Sep 08 13:55:16.512351 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.512319    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="92d05aef-6bc0-47ae-8abc-ab11a6416c2d" containerName="main"
Sep 08 13:55:16.512441 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.512418    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="92d05aef-6bc0-47ae-8abc-ab11a6416c2d" containerName="main"
Sep 08 13:55:16.512441 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.512430    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9864420c-5be6-442d-9381-79db08992db3" containerName="main"
Sep 08 13:55:16.512441 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.512437    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a180d75f-2f98-4045-b642-7acd13bc94d4" containerName="main"
Sep 08 13:55:16.515344 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.515323    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs"
Sep 08 13:55:16.517859 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.517839    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 08 13:55:16.518504 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.518481    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 08 13:55:16.518639 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.518566    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 08 13:55:16.518639 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.518567    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-8dkz9\""
Sep 08 13:55:16.523626 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.523250    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs"]
Sep 08 13:55:16.664867 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.664827    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs\" (UID: \"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs"
Sep 08 13:55:16.665080 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.664890    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs\" (UID: \"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs"
Sep 08 13:55:16.665080 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.664967    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqrt5\" (UniqueName: \"kubernetes.io/projected/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-kube-api-access-nqrt5\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs\" (UID: \"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs"
Sep 08 13:55:16.665080 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.664999    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs\" (UID: \"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs"
Sep 08 13:55:16.665080 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.665023    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs\" (UID: \"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs"
Sep 08 13:55:16.665243 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.665108    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs\" (UID: \"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs"
Sep 08 13:55:16.766484 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.766401    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqrt5\" (UniqueName: \"kubernetes.io/projected/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-kube-api-access-nqrt5\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs\" (UID: \"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs"
Sep 08 13:55:16.766484 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.766453    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs\" (UID: \"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs"
Sep 08 13:55:16.766484 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.766483    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs\" (UID: \"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs"
Sep 08 13:55:16.766756 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.766544    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs\" (UID: \"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs"
Sep 08 13:55:16.766756 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.766639    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs\" (UID: \"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs"
Sep 08 13:55:16.766756 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.766704    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs\" (UID: \"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs"
Sep 08 13:55:16.766944 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.766923    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs\" (UID: \"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs"
Sep 08 13:55:16.767002 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.766964    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs\" (UID: \"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs"
Sep 08 13:55:16.767095 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.767076    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs\" (UID: \"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs"
Sep 08 13:55:16.768832 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.768813    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs\" (UID: \"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs"
Sep 08 13:55:16.769193 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.769170    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs\" (UID: \"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs"
Sep 08 13:55:16.776412 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.776386    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqrt5\" (UniqueName: \"kubernetes.io/projected/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-kube-api-access-nqrt5\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs\" (UID: \"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs"
Sep 08 13:55:16.828112 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:16.828070    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs"
Sep 08 13:55:17.106438 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:17.106410    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs"]
Sep 08 13:55:17.108293 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:55:17.108260    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b4afe5_4acb_44aa_8bf8_1e3ebeefa70f.slice/crio-4467ea66fc8281a7a686a4e7775c0f492099a22b6f7f16d2c3a2582871900b14 WatchSource:0}: Error finding container 4467ea66fc8281a7a686a4e7775c0f492099a22b6f7f16d2c3a2582871900b14: Status 404 returned error can't find the container with id 4467ea66fc8281a7a686a4e7775c0f492099a22b6f7f16d2c3a2582871900b14
Sep 08 13:55:17.532828 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:17.532791    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs" event={"ID":"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f","Type":"ContainerStarted","Data":"e7a12ef509483ae3d9bd8f5560eb437ae9095f5169a354d7ee957739f2545bd8"}
Sep 08 13:55:17.533310 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:17.532841    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs" event={"ID":"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f","Type":"ContainerStarted","Data":"4467ea66fc8281a7a686a4e7775c0f492099a22b6f7f16d2c3a2582871900b14"}
Sep 08 13:55:17.577724 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:17.577665    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs" podStartSLOduration=1.5776459470000002 podStartE2EDuration="1.577645947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:55:17.574822159 +0000 UTC m=+2381.026583091" watchObservedRunningTime="2026-09-08 13:55:17.577645947 +0000 UTC m=+2381.029406877"
Sep 08 13:55:17.828835 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:17.828744    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs"
Sep 08 13:55:26.828541 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:26.828493    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs"
Sep 08 13:55:26.841194 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:26.841160    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs"
Sep 08 13:55:27.590510 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:55:27.590481    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs"
Sep 08 13:56:04.058371 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:04.058279    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs"]
Sep 08 13:56:04.067986 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:56:04.067947    2558 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 08 13:56:04.067986 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:56:04.067986 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:56:04.067986 ip-10-0-128-147 kubenswrapper[2558]:  > containerID="e7a12ef509483ae3d9bd8f5560eb437ae9095f5169a354d7ee957739f2545bd8" cmd=["/bin/sleep","15"]
Sep 08 13:56:04.068236 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:04.068003    2558 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs" message="" err=<
Sep 08 13:56:04.068236 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:56:04.068236 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:56:04.068236 ip-10-0-128-147 kubenswrapper[2558]:  >
Sep 08 13:56:04.068236 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:56:04.068019    2558 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 08 13:56:04.068236 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:56:04.068236 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:56:04.068236 ip-10-0-128-147 kubenswrapper[2558]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs" podUID="30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f" containerName="main" containerID="cri-o://e7a12ef509483ae3d9bd8f5560eb437ae9095f5169a354d7ee957739f2545bd8"
Sep 08 13:56:04.068236 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:04.068070    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs" podUID="30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f" containerName="main" containerID="cri-o://e7a12ef509483ae3d9bd8f5560eb437ae9095f5169a354d7ee957739f2545bd8" gracePeriod=60
Sep 08 13:56:04.419244 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:04.419219    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs"
Sep 08 13:56:04.589715 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:04.589617    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-tmp-dir\") pod \"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f\" (UID: \"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f\") "
Sep 08 13:56:04.589715 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:04.589657    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-home\") pod \"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f\" (UID: \"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f\") "
Sep 08 13:56:04.589715 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:04.589690    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqrt5\" (UniqueName: \"kubernetes.io/projected/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-kube-api-access-nqrt5\") pod \"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f\" (UID: \"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f\") "
Sep 08 13:56:04.590010 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:04.589722    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-model-cache\") pod \"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f\" (UID: \"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f\") "
Sep 08 13:56:04.590010 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:04.589883    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-dshm\") pod \"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f\" (UID: \"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f\") "
Sep 08 13:56:04.590010 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:04.589927    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-tls-certs\") pod \"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f\" (UID: \"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f\") "
Sep 08 13:56:04.590010 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:04.589880    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f" (UID: "30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:56:04.590173 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:04.590009    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-home" (OuterVolumeSpecName: "home") pod "30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f" (UID: "30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:56:04.590173 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:04.590032    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-model-cache" (OuterVolumeSpecName: "model-cache") pod "30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f" (UID: "30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:56:04.590239 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:04.590214    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:56:04.590281 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:04.590236    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:56:04.590281 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:04.590251    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:56:04.592073 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:04.592042    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-dshm" (OuterVolumeSpecName: "dshm") pod "30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f" (UID: "30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:56:04.592073 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:04.592044    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f" (UID: "30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:56:04.592222 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:04.592107    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-kube-api-access-nqrt5" (OuterVolumeSpecName: "kube-api-access-nqrt5") pod "30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f" (UID: "30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f"). InnerVolumeSpecName "kube-api-access-nqrt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:56:04.691144 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:04.691104    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:56:04.691144 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:04.691136    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:56:04.691144 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:04.691148    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqrt5\" (UniqueName: \"kubernetes.io/projected/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f-kube-api-access-nqrt5\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:56:04.765994 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:04.765954    2558 generic.go:356] "Generic (PLEG): container finished" podID="30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f" containerID="e7a12ef509483ae3d9bd8f5560eb437ae9095f5169a354d7ee957739f2545bd8" exitCode=0
Sep 08 13:56:04.766203 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:04.766005    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs"
Sep 08 13:56:04.766203 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:04.766030    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs" event={"ID":"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f","Type":"ContainerDied","Data":"e7a12ef509483ae3d9bd8f5560eb437ae9095f5169a354d7ee957739f2545bd8"}
Sep 08 13:56:04.766203 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:04.766104    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs" event={"ID":"30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f","Type":"ContainerDied","Data":"4467ea66fc8281a7a686a4e7775c0f492099a22b6f7f16d2c3a2582871900b14"}
Sep 08 13:56:04.766203 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:04.766135    2558 scope.go:117] "RemoveContainer" containerID="e7a12ef509483ae3d9bd8f5560eb437ae9095f5169a354d7ee957739f2545bd8"
Sep 08 13:56:04.775951 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:04.775932    2558 scope.go:117] "RemoveContainer" containerID="e7a12ef509483ae3d9bd8f5560eb437ae9095f5169a354d7ee957739f2545bd8"
Sep 08 13:56:04.776210 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:56:04.776190    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7a12ef509483ae3d9bd8f5560eb437ae9095f5169a354d7ee957739f2545bd8\": container with ID starting with e7a12ef509483ae3d9bd8f5560eb437ae9095f5169a354d7ee957739f2545bd8 not found: ID does not exist" containerID="e7a12ef509483ae3d9bd8f5560eb437ae9095f5169a354d7ee957739f2545bd8"
Sep 08 13:56:04.776288 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:04.776218    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7a12ef509483ae3d9bd8f5560eb437ae9095f5169a354d7ee957739f2545bd8"} err="failed to get container status \"e7a12ef509483ae3d9bd8f5560eb437ae9095f5169a354d7ee957739f2545bd8\": rpc error: code = NotFound desc = could not find container \"e7a12ef509483ae3d9bd8f5560eb437ae9095f5169a354d7ee957739f2545bd8\": container with ID starting with e7a12ef509483ae3d9bd8f5560eb437ae9095f5169a354d7ee957739f2545bd8 not found: ID does not exist"
Sep 08 13:56:04.788977 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:04.788953    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs"]
Sep 08 13:56:04.795124 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:04.795101    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-c2cqs"]
Sep 08 13:56:05.054385 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:05.054354    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f" path="/var/lib/kubelet/pods/30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f/volumes"
Sep 08 13:56:31.986805 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:31.986764    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9"]
Sep 08 13:56:45.325150 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:45.325113    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn"]
Sep 08 13:56:45.325535 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:45.325484    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f" containerName="main"
Sep 08 13:56:45.325535 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:45.325496    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f" containerName="main"
Sep 08 13:56:45.325627 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:45.325565    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="30b4afe5-4acb-44aa-8bf8-1e3ebeefa70f" containerName="main"
Sep 08 13:56:45.328440 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:45.328423    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn"
Sep 08 13:56:45.334694 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:45.334666    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"kube-root-ca.crt\""
Sep 08 13:56:45.335698 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:45.335679    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"default-dockercfg-nksjg\""
Sep 08 13:56:45.335814 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:45.335698    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"openshift-service-ca.crt\""
Sep 08 13:56:45.335814 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:45.335714    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v1-kserve-self-signed-certs\""
Sep 08 13:56:45.361261 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:45.361223    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn"]
Sep 08 13:56:45.455560 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:45.455521    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61432cd7-478e-4671-b56f-57f382f5050a-dshm\") pod \"delw-v1-kserve-6cf8cbb847-996wn\" (UID: \"61432cd7-478e-4671-b56f-57f382f5050a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn"
Sep 08 13:56:45.455791 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:45.455581    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61432cd7-478e-4671-b56f-57f382f5050a-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-996wn\" (UID: \"61432cd7-478e-4671-b56f-57f382f5050a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn"
Sep 08 13:56:45.455791 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:45.455659    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bxdw\" (UniqueName: \"kubernetes.io/projected/61432cd7-478e-4671-b56f-57f382f5050a-kube-api-access-6bxdw\") pod \"delw-v1-kserve-6cf8cbb847-996wn\" (UID: \"61432cd7-478e-4671-b56f-57f382f5050a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn"
Sep 08 13:56:45.455791 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:45.455717    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61432cd7-478e-4671-b56f-57f382f5050a-home\") pod \"delw-v1-kserve-6cf8cbb847-996wn\" (UID: \"61432cd7-478e-4671-b56f-57f382f5050a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn"
Sep 08 13:56:45.455791 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:45.455749    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61432cd7-478e-4671-b56f-57f382f5050a-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-996wn\" (UID: \"61432cd7-478e-4671-b56f-57f382f5050a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn"
Sep 08 13:56:45.455971 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:45.455793    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61432cd7-478e-4671-b56f-57f382f5050a-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-996wn\" (UID: \"61432cd7-478e-4671-b56f-57f382f5050a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn"
Sep 08 13:56:45.556733 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:45.556693    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61432cd7-478e-4671-b56f-57f382f5050a-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-996wn\" (UID: \"61432cd7-478e-4671-b56f-57f382f5050a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn"
Sep 08 13:56:45.556930 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:45.556749    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bxdw\" (UniqueName: \"kubernetes.io/projected/61432cd7-478e-4671-b56f-57f382f5050a-kube-api-access-6bxdw\") pod \"delw-v1-kserve-6cf8cbb847-996wn\" (UID: \"61432cd7-478e-4671-b56f-57f382f5050a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn"
Sep 08 13:56:45.556930 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:45.556795    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61432cd7-478e-4671-b56f-57f382f5050a-home\") pod \"delw-v1-kserve-6cf8cbb847-996wn\" (UID: \"61432cd7-478e-4671-b56f-57f382f5050a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn"
Sep 08 13:56:45.556930 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:45.556828    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61432cd7-478e-4671-b56f-57f382f5050a-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-996wn\" (UID: \"61432cd7-478e-4671-b56f-57f382f5050a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn"
Sep 08 13:56:45.556930 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:45.556861    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61432cd7-478e-4671-b56f-57f382f5050a-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-996wn\" (UID: \"61432cd7-478e-4671-b56f-57f382f5050a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn"
Sep 08 13:56:45.556930 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:45.556911    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61432cd7-478e-4671-b56f-57f382f5050a-dshm\") pod \"delw-v1-kserve-6cf8cbb847-996wn\" (UID: \"61432cd7-478e-4671-b56f-57f382f5050a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn"
Sep 08 13:56:45.557203 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:45.557145    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61432cd7-478e-4671-b56f-57f382f5050a-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-996wn\" (UID: \"61432cd7-478e-4671-b56f-57f382f5050a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn"
Sep 08 13:56:45.557203 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:45.557176    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61432cd7-478e-4671-b56f-57f382f5050a-home\") pod \"delw-v1-kserve-6cf8cbb847-996wn\" (UID: \"61432cd7-478e-4671-b56f-57f382f5050a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn"
Sep 08 13:56:45.557310 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:45.557219    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61432cd7-478e-4671-b56f-57f382f5050a-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-996wn\" (UID: \"61432cd7-478e-4671-b56f-57f382f5050a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn"
Sep 08 13:56:45.559155 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:45.559134    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61432cd7-478e-4671-b56f-57f382f5050a-dshm\") pod \"delw-v1-kserve-6cf8cbb847-996wn\" (UID: \"61432cd7-478e-4671-b56f-57f382f5050a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn"
Sep 08 13:56:45.559421 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:45.559404    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61432cd7-478e-4671-b56f-57f382f5050a-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-996wn\" (UID: \"61432cd7-478e-4671-b56f-57f382f5050a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn"
Sep 08 13:56:45.565335 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:45.565312    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bxdw\" (UniqueName: \"kubernetes.io/projected/61432cd7-478e-4671-b56f-57f382f5050a-kube-api-access-6bxdw\") pod \"delw-v1-kserve-6cf8cbb847-996wn\" (UID: \"61432cd7-478e-4671-b56f-57f382f5050a\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn"
Sep 08 13:56:45.638747 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:45.638715    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn"
Sep 08 13:56:45.928064 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:45.928034    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn"]
Sep 08 13:56:45.930318 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:56:45.930284    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61432cd7_478e_4671_b56f_57f382f5050a.slice/crio-04dcbfb914e6e3dbed10b5a357d5ffdf8599b6f90089884f6d5304382a80c480 WatchSource:0}: Error finding container 04dcbfb914e6e3dbed10b5a357d5ffdf8599b6f90089884f6d5304382a80c480: Status 404 returned error can't find the container with id 04dcbfb914e6e3dbed10b5a357d5ffdf8599b6f90089884f6d5304382a80c480
Sep 08 13:56:45.965903 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:45.965877    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn" event={"ID":"61432cd7-478e-4671-b56f-57f382f5050a","Type":"ContainerStarted","Data":"04dcbfb914e6e3dbed10b5a357d5ffdf8599b6f90089884f6d5304382a80c480"}
Sep 08 13:56:46.971733 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:46.971688    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn" event={"ID":"61432cd7-478e-4671-b56f-57f382f5050a","Type":"ContainerStarted","Data":"f664d21c73722a847f7bcdaa105e2189d8773a2c9eeb5e599dc9c9246d5de297"}
Sep 08 13:56:47.006232 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:47.006195    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9" podUID="2e682304-c831-4b80-b597-597383c3a776" containerName="main" containerID="cri-o://3e3075dc8714c31fa80abd975d7b72fea8beb2c087037d81b18b62b6b8c6bc09" gracePeriod=45
Sep 08 13:56:47.033640 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:47.030608    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn" podStartSLOduration=2.030569567 podStartE2EDuration="2.030569567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:56:47.027159379 +0000 UTC m=+2470.478920315" watchObservedRunningTime="2026-09-08 13:56:47.030569567 +0000 UTC m=+2470.482330499"
Sep 08 13:56:47.254359 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:47.254334    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9"
Sep 08 13:56:47.372462 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:47.372433    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e682304-c831-4b80-b597-597383c3a776-model-cache\") pod \"2e682304-c831-4b80-b597-597383c3a776\" (UID: \"2e682304-c831-4b80-b597-597383c3a776\") "
Sep 08 13:56:47.372688 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:47.372483    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e682304-c831-4b80-b597-597383c3a776-tmp-dir\") pod \"2e682304-c831-4b80-b597-597383c3a776\" (UID: \"2e682304-c831-4b80-b597-597383c3a776\") "
Sep 08 13:56:47.372688 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:47.372545    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdltq\" (UniqueName: \"kubernetes.io/projected/2e682304-c831-4b80-b597-597383c3a776-kube-api-access-vdltq\") pod \"2e682304-c831-4b80-b597-597383c3a776\" (UID: \"2e682304-c831-4b80-b597-597383c3a776\") "
Sep 08 13:56:47.372688 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:47.372571    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e682304-c831-4b80-b597-597383c3a776-dshm\") pod \"2e682304-c831-4b80-b597-597383c3a776\" (UID: \"2e682304-c831-4b80-b597-597383c3a776\") "
Sep 08 13:56:47.372688 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:47.372642    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e682304-c831-4b80-b597-597383c3a776-tls-certs\") pod \"2e682304-c831-4b80-b597-597383c3a776\" (UID: \"2e682304-c831-4b80-b597-597383c3a776\") "
Sep 08 13:56:47.372912 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:47.372692    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e682304-c831-4b80-b597-597383c3a776-home\") pod \"2e682304-c831-4b80-b597-597383c3a776\" (UID: \"2e682304-c831-4b80-b597-597383c3a776\") "
Sep 08 13:56:47.372912 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:47.372726    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e682304-c831-4b80-b597-597383c3a776-model-cache" (OuterVolumeSpecName: "model-cache") pod "2e682304-c831-4b80-b597-597383c3a776" (UID: "2e682304-c831-4b80-b597-597383c3a776"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:56:47.372912 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:47.372814    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e682304-c831-4b80-b597-597383c3a776-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2e682304-c831-4b80-b597-597383c3a776" (UID: "2e682304-c831-4b80-b597-597383c3a776"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:56:47.373067 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:47.372977    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e682304-c831-4b80-b597-597383c3a776-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:56:47.373067 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:47.372991    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e682304-c831-4b80-b597-597383c3a776-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:56:47.373067 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:47.372970    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e682304-c831-4b80-b597-597383c3a776-home" (OuterVolumeSpecName: "home") pod "2e682304-c831-4b80-b597-597383c3a776" (UID: "2e682304-c831-4b80-b597-597383c3a776"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:56:47.374833 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:47.374804    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e682304-c831-4b80-b597-597383c3a776-kube-api-access-vdltq" (OuterVolumeSpecName: "kube-api-access-vdltq") pod "2e682304-c831-4b80-b597-597383c3a776" (UID: "2e682304-c831-4b80-b597-597383c3a776"). InnerVolumeSpecName "kube-api-access-vdltq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:56:47.374979 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:47.374961    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e682304-c831-4b80-b597-597383c3a776-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e682304-c831-4b80-b597-597383c3a776" (UID: "2e682304-c831-4b80-b597-597383c3a776"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:56:47.375022 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:47.374975    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e682304-c831-4b80-b597-597383c3a776-dshm" (OuterVolumeSpecName: "dshm") pod "2e682304-c831-4b80-b597-597383c3a776" (UID: "2e682304-c831-4b80-b597-597383c3a776"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:56:47.473726 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:47.473698    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdltq\" (UniqueName: \"kubernetes.io/projected/2e682304-c831-4b80-b597-597383c3a776-kube-api-access-vdltq\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:56:47.473726 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:47.473722    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e682304-c831-4b80-b597-597383c3a776-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:56:47.473887 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:47.473733    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e682304-c831-4b80-b597-597383c3a776-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:56:47.473887 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:47.473742    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e682304-c831-4b80-b597-597383c3a776-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:56:47.639343 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:47.639313    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn"
Sep 08 13:56:47.977501 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:47.977416    2558 generic.go:356] "Generic (PLEG): container finished" podID="2e682304-c831-4b80-b597-597383c3a776" containerID="3e3075dc8714c31fa80abd975d7b72fea8beb2c087037d81b18b62b6b8c6bc09" exitCode=0
Sep 08 13:56:47.977501 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:47.977461    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9"
Sep 08 13:56:47.978032 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:47.977498    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9" event={"ID":"2e682304-c831-4b80-b597-597383c3a776","Type":"ContainerDied","Data":"3e3075dc8714c31fa80abd975d7b72fea8beb2c087037d81b18b62b6b8c6bc09"}
Sep 08 13:56:47.978032 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:47.977553    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9" event={"ID":"2e682304-c831-4b80-b597-597383c3a776","Type":"ContainerDied","Data":"268e9621b8a15c74311142c8435e1e1cefded08d433288ffd74796a487907e15"}
Sep 08 13:56:47.978032 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:47.977585    2558 scope.go:117] "RemoveContainer" containerID="3e3075dc8714c31fa80abd975d7b72fea8beb2c087037d81b18b62b6b8c6bc09"
Sep 08 13:56:47.986258 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:47.986241    2558 scope.go:117] "RemoveContainer" containerID="3e3075dc8714c31fa80abd975d7b72fea8beb2c087037d81b18b62b6b8c6bc09"
Sep 08 13:56:47.986512 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:56:47.986494    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e3075dc8714c31fa80abd975d7b72fea8beb2c087037d81b18b62b6b8c6bc09\": container with ID starting with 3e3075dc8714c31fa80abd975d7b72fea8beb2c087037d81b18b62b6b8c6bc09 not found: ID does not exist" containerID="3e3075dc8714c31fa80abd975d7b72fea8beb2c087037d81b18b62b6b8c6bc09"
Sep 08 13:56:47.986562 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:47.986520    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e3075dc8714c31fa80abd975d7b72fea8beb2c087037d81b18b62b6b8c6bc09"} err="failed to get container status \"3e3075dc8714c31fa80abd975d7b72fea8beb2c087037d81b18b62b6b8c6bc09\": rpc error: code = NotFound desc = could not find container \"3e3075dc8714c31fa80abd975d7b72fea8beb2c087037d81b18b62b6b8c6bc09\": container with ID starting with 3e3075dc8714c31fa80abd975d7b72fea8beb2c087037d81b18b62b6b8c6bc09 not found: ID does not exist"
Sep 08 13:56:48.002490 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:48.002471    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9"]
Sep 08 13:56:48.011506 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:48.011485    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-pq6j9"]
Sep 08 13:56:49.053480 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:49.053444    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e682304-c831-4b80-b597-597383c3a776" path="/var/lib/kubelet/pods/2e682304-c831-4b80-b597-597383c3a776/volumes"
Sep 08 13:56:55.639248 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:55.639210    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn"
Sep 08 13:56:55.651879 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:55.651855    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn"
Sep 08 13:56:56.026746 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:56:56.026658    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn"
Sep 08 13:57:44.010482 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:44.010444    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb"]
Sep 08 13:57:44.010928 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:44.010824    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e682304-c831-4b80-b597-597383c3a776" containerName="main"
Sep 08 13:57:44.010928 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:44.010835    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e682304-c831-4b80-b597-597383c3a776" containerName="main"
Sep 08 13:57:44.010928 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:44.010903    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e682304-c831-4b80-b597-597383c3a776" containerName="main"
Sep 08 13:57:44.015339 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:44.015317    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb"
Sep 08 13:57:44.018668 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:44.018643    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 08 13:57:44.018813 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:44.018646    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 08 13:57:44.018885 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:44.018692    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-l7glm\""
Sep 08 13:57:44.018962 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:44.018702    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 08 13:57:44.025947 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:44.025916    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb"]
Sep 08 13:57:44.155005 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:44.154959    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4790bad-6605-422b-bdf7-30d69b575c4e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb\" (UID: \"a4790bad-6605-422b-bdf7-30d69b575c4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb"
Sep 08 13:57:44.155005 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:44.155008    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4790bad-6605-422b-bdf7-30d69b575c4e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb\" (UID: \"a4790bad-6605-422b-bdf7-30d69b575c4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb"
Sep 08 13:57:44.155229 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:44.155073    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz5vx\" (UniqueName: \"kubernetes.io/projected/a4790bad-6605-422b-bdf7-30d69b575c4e-kube-api-access-sz5vx\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb\" (UID: \"a4790bad-6605-422b-bdf7-30d69b575c4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb"
Sep 08 13:57:44.155229 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:44.155150    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4790bad-6605-422b-bdf7-30d69b575c4e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb\" (UID: \"a4790bad-6605-422b-bdf7-30d69b575c4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb"
Sep 08 13:57:44.155229 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:44.155210    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4790bad-6605-422b-bdf7-30d69b575c4e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb\" (UID: \"a4790bad-6605-422b-bdf7-30d69b575c4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb"
Sep 08 13:57:44.155229 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:44.155225    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4790bad-6605-422b-bdf7-30d69b575c4e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb\" (UID: \"a4790bad-6605-422b-bdf7-30d69b575c4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb"
Sep 08 13:57:44.255937 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:44.255903    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4790bad-6605-422b-bdf7-30d69b575c4e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb\" (UID: \"a4790bad-6605-422b-bdf7-30d69b575c4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb"
Sep 08 13:57:44.255937 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:44.255938    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4790bad-6605-422b-bdf7-30d69b575c4e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb\" (UID: \"a4790bad-6605-422b-bdf7-30d69b575c4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb"
Sep 08 13:57:44.256200 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:44.255958    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz5vx\" (UniqueName: \"kubernetes.io/projected/a4790bad-6605-422b-bdf7-30d69b575c4e-kube-api-access-sz5vx\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb\" (UID: \"a4790bad-6605-422b-bdf7-30d69b575c4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb"
Sep 08 13:57:44.256200 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:44.255986    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4790bad-6605-422b-bdf7-30d69b575c4e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb\" (UID: \"a4790bad-6605-422b-bdf7-30d69b575c4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb"
Sep 08 13:57:44.256200 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:44.256041    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4790bad-6605-422b-bdf7-30d69b575c4e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb\" (UID: \"a4790bad-6605-422b-bdf7-30d69b575c4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb"
Sep 08 13:57:44.256200 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:44.256066    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4790bad-6605-422b-bdf7-30d69b575c4e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb\" (UID: \"a4790bad-6605-422b-bdf7-30d69b575c4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb"
Sep 08 13:57:44.256425 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:44.256395    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4790bad-6605-422b-bdf7-30d69b575c4e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb\" (UID: \"a4790bad-6605-422b-bdf7-30d69b575c4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb"
Sep 08 13:57:44.256482 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:44.256448    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4790bad-6605-422b-bdf7-30d69b575c4e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb\" (UID: \"a4790bad-6605-422b-bdf7-30d69b575c4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb"
Sep 08 13:57:44.256528 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:44.256480    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4790bad-6605-422b-bdf7-30d69b575c4e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb\" (UID: \"a4790bad-6605-422b-bdf7-30d69b575c4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb"
Sep 08 13:57:44.258363 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:44.258334    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4790bad-6605-422b-bdf7-30d69b575c4e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb\" (UID: \"a4790bad-6605-422b-bdf7-30d69b575c4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb"
Sep 08 13:57:44.258666 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:44.258644    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4790bad-6605-422b-bdf7-30d69b575c4e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb\" (UID: \"a4790bad-6605-422b-bdf7-30d69b575c4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb"
Sep 08 13:57:44.264714 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:44.264667    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz5vx\" (UniqueName: \"kubernetes.io/projected/a4790bad-6605-422b-bdf7-30d69b575c4e-kube-api-access-sz5vx\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb\" (UID: \"a4790bad-6605-422b-bdf7-30d69b575c4e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb"
Sep 08 13:57:44.327796 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:44.327754    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb"
Sep 08 13:57:44.607297 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:44.607271    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb"]
Sep 08 13:57:44.609604 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:57:44.609564    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4790bad_6605_422b_bdf7_30d69b575c4e.slice/crio-4b0693ae1a2d0fe54e2c4337b3ad09915e62649e354d3bf95854ac3e4c1157d8 WatchSource:0}: Error finding container 4b0693ae1a2d0fe54e2c4337b3ad09915e62649e354d3bf95854ac3e4c1157d8: Status 404 returned error can't find the container with id 4b0693ae1a2d0fe54e2c4337b3ad09915e62649e354d3bf95854ac3e4c1157d8
Sep 08 13:57:45.255043 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:45.255007    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb" event={"ID":"a4790bad-6605-422b-bdf7-30d69b575c4e","Type":"ContainerStarted","Data":"815a2d5157b2db7a26562e391b344ae7b9bbb9fe23271070736a9d9e0a417651"}
Sep 08 13:57:45.255501 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:45.255060    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb" event={"ID":"a4790bad-6605-422b-bdf7-30d69b575c4e","Type":"ContainerStarted","Data":"4b0693ae1a2d0fe54e2c4337b3ad09915e62649e354d3bf95854ac3e4c1157d8"}
Sep 08 13:57:45.280203 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:45.280153    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb" podStartSLOduration=2.280136663 podStartE2EDuration="2.280136663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:57:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:57:45.278763206 +0000 UTC m=+2528.730524138" watchObservedRunningTime="2026-09-08 13:57:45.280136663 +0000 UTC m=+2528.731897593"
Sep 08 13:57:45.327928 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:45.327894    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb"
Sep 08 13:57:54.328383 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:54.328347    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb"
Sep 08 13:57:54.340767 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:54.340739    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb"
Sep 08 13:57:55.316049 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:57:55.316019    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb"
Sep 08 13:58:10.424660 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:10.424618    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn"]
Sep 08 13:58:25.444670 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:25.444631    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn" podUID="61432cd7-478e-4671-b56f-57f382f5050a" containerName="main" containerID="cri-o://f664d21c73722a847f7bcdaa105e2189d8773a2c9eeb5e599dc9c9246d5de297" gracePeriod=45
Sep 08 13:58:25.694968 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:25.694902    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn"
Sep 08 13:58:25.823770 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:25.823731    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61432cd7-478e-4671-b56f-57f382f5050a-dshm\") pod \"61432cd7-478e-4671-b56f-57f382f5050a\" (UID: \"61432cd7-478e-4671-b56f-57f382f5050a\") "
Sep 08 13:58:25.823954 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:25.823827    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61432cd7-478e-4671-b56f-57f382f5050a-model-cache\") pod \"61432cd7-478e-4671-b56f-57f382f5050a\" (UID: \"61432cd7-478e-4671-b56f-57f382f5050a\") "
Sep 08 13:58:25.823954 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:25.823884    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61432cd7-478e-4671-b56f-57f382f5050a-tls-certs\") pod \"61432cd7-478e-4671-b56f-57f382f5050a\" (UID: \"61432cd7-478e-4671-b56f-57f382f5050a\") "
Sep 08 13:58:25.823954 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:25.823910    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61432cd7-478e-4671-b56f-57f382f5050a-tmp-dir\") pod \"61432cd7-478e-4671-b56f-57f382f5050a\" (UID: \"61432cd7-478e-4671-b56f-57f382f5050a\") "
Sep 08 13:58:25.823954 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:25.823942    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61432cd7-478e-4671-b56f-57f382f5050a-home\") pod \"61432cd7-478e-4671-b56f-57f382f5050a\" (UID: \"61432cd7-478e-4671-b56f-57f382f5050a\") "
Sep 08 13:58:25.824177 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:25.824001    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bxdw\" (UniqueName: \"kubernetes.io/projected/61432cd7-478e-4671-b56f-57f382f5050a-kube-api-access-6bxdw\") pod \"61432cd7-478e-4671-b56f-57f382f5050a\" (UID: \"61432cd7-478e-4671-b56f-57f382f5050a\") "
Sep 08 13:58:25.824177 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:25.824107    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61432cd7-478e-4671-b56f-57f382f5050a-model-cache" (OuterVolumeSpecName: "model-cache") pod "61432cd7-478e-4671-b56f-57f382f5050a" (UID: "61432cd7-478e-4671-b56f-57f382f5050a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:58:25.824287 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:25.824228    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61432cd7-478e-4671-b56f-57f382f5050a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "61432cd7-478e-4671-b56f-57f382f5050a" (UID: "61432cd7-478e-4671-b56f-57f382f5050a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:58:25.824287 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:25.824240    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61432cd7-478e-4671-b56f-57f382f5050a-home" (OuterVolumeSpecName: "home") pod "61432cd7-478e-4671-b56f-57f382f5050a" (UID: "61432cd7-478e-4671-b56f-57f382f5050a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:58:25.824415 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:25.824338    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61432cd7-478e-4671-b56f-57f382f5050a-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:58:25.824415 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:25.824351    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61432cd7-478e-4671-b56f-57f382f5050a-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:58:25.824415 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:25.824359    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61432cd7-478e-4671-b56f-57f382f5050a-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:58:25.826063 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:25.826042    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61432cd7-478e-4671-b56f-57f382f5050a-dshm" (OuterVolumeSpecName: "dshm") pod "61432cd7-478e-4671-b56f-57f382f5050a" (UID: "61432cd7-478e-4671-b56f-57f382f5050a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:58:25.826169 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:25.826144    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61432cd7-478e-4671-b56f-57f382f5050a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "61432cd7-478e-4671-b56f-57f382f5050a" (UID: "61432cd7-478e-4671-b56f-57f382f5050a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:58:25.826242 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:25.826218    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61432cd7-478e-4671-b56f-57f382f5050a-kube-api-access-6bxdw" (OuterVolumeSpecName: "kube-api-access-6bxdw") pod "61432cd7-478e-4671-b56f-57f382f5050a" (UID: "61432cd7-478e-4671-b56f-57f382f5050a"). InnerVolumeSpecName "kube-api-access-6bxdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:58:25.925334 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:25.925293    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61432cd7-478e-4671-b56f-57f382f5050a-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:58:25.925334 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:25.925325    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bxdw\" (UniqueName: \"kubernetes.io/projected/61432cd7-478e-4671-b56f-57f382f5050a-kube-api-access-6bxdw\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:58:25.925334 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:25.925335    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61432cd7-478e-4671-b56f-57f382f5050a-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:58:26.456322 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:26.456284    2558 generic.go:356] "Generic (PLEG): container finished" podID="61432cd7-478e-4671-b56f-57f382f5050a" containerID="f664d21c73722a847f7bcdaa105e2189d8773a2c9eeb5e599dc9c9246d5de297" exitCode=0
Sep 08 13:58:26.456821 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:26.456329    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn"
Sep 08 13:58:26.456821 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:26.456370    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn" event={"ID":"61432cd7-478e-4671-b56f-57f382f5050a","Type":"ContainerDied","Data":"f664d21c73722a847f7bcdaa105e2189d8773a2c9eeb5e599dc9c9246d5de297"}
Sep 08 13:58:26.456821 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:26.456426    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn" event={"ID":"61432cd7-478e-4671-b56f-57f382f5050a","Type":"ContainerDied","Data":"04dcbfb914e6e3dbed10b5a357d5ffdf8599b6f90089884f6d5304382a80c480"}
Sep 08 13:58:26.456821 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:26.456455    2558 scope.go:117] "RemoveContainer" containerID="f664d21c73722a847f7bcdaa105e2189d8773a2c9eeb5e599dc9c9246d5de297"
Sep 08 13:58:26.466117 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:26.466097    2558 scope.go:117] "RemoveContainer" containerID="f664d21c73722a847f7bcdaa105e2189d8773a2c9eeb5e599dc9c9246d5de297"
Sep 08 13:58:26.466422 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:58:26.466393    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f664d21c73722a847f7bcdaa105e2189d8773a2c9eeb5e599dc9c9246d5de297\": container with ID starting with f664d21c73722a847f7bcdaa105e2189d8773a2c9eeb5e599dc9c9246d5de297 not found: ID does not exist" containerID="f664d21c73722a847f7bcdaa105e2189d8773a2c9eeb5e599dc9c9246d5de297"
Sep 08 13:58:26.466485 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:26.466435    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f664d21c73722a847f7bcdaa105e2189d8773a2c9eeb5e599dc9c9246d5de297"} err="failed to get container status \"f664d21c73722a847f7bcdaa105e2189d8773a2c9eeb5e599dc9c9246d5de297\": rpc error: code = NotFound desc = could not find container \"f664d21c73722a847f7bcdaa105e2189d8773a2c9eeb5e599dc9c9246d5de297\": container with ID starting with f664d21c73722a847f7bcdaa105e2189d8773a2c9eeb5e599dc9c9246d5de297 not found: ID does not exist"
Sep 08 13:58:26.480151 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:26.480119    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn"]
Sep 08 13:58:26.485934 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:26.485909    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-996wn"]
Sep 08 13:58:27.059752 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:27.059714    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61432cd7-478e-4671-b56f-57f382f5050a" path="/var/lib/kubelet/pods/61432cd7-478e-4671-b56f-57f382f5050a/volumes"
Sep 08 13:58:40.505044 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:40.505012    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64"]
Sep 08 13:58:40.505513 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:40.505476    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61432cd7-478e-4671-b56f-57f382f5050a" containerName="main"
Sep 08 13:58:40.505513 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:40.505489    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="61432cd7-478e-4671-b56f-57f382f5050a" containerName="main"
Sep 08 13:58:40.505671 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:40.505553    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="61432cd7-478e-4671-b56f-57f382f5050a" containerName="main"
Sep 08 13:58:40.510419 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:40.510399    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64"
Sep 08 13:58:40.513565 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:40.513532    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"kube-root-ca.crt\""
Sep 08 13:58:40.513565 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:40.513551    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v1-kserve-self-signed-certs\""
Sep 08 13:58:40.513762 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:40.513537    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"openshift-service-ca.crt\""
Sep 08 13:58:40.513762 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:40.513605    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"default-dockercfg-g727d\""
Sep 08 13:58:40.516934 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:40.516912    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64"]
Sep 08 13:58:40.546062 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:40.546032    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adb91d40-1c23-456d-afa0-aa6b137e5c0a-home\") pod \"roll-v1-kserve-69cf8bcfc-msh64\" (UID: \"adb91d40-1c23-456d-afa0-aa6b137e5c0a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64"
Sep 08 13:58:40.546062 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:40.546066    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adb91d40-1c23-456d-afa0-aa6b137e5c0a-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-msh64\" (UID: \"adb91d40-1c23-456d-afa0-aa6b137e5c0a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64"
Sep 08 13:58:40.546263 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:40.546094    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adb91d40-1c23-456d-afa0-aa6b137e5c0a-dshm\") pod \"roll-v1-kserve-69cf8bcfc-msh64\" (UID: \"adb91d40-1c23-456d-afa0-aa6b137e5c0a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64"
Sep 08 13:58:40.546263 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:40.546151    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhhwv\" (UniqueName: \"kubernetes.io/projected/adb91d40-1c23-456d-afa0-aa6b137e5c0a-kube-api-access-zhhwv\") pod \"roll-v1-kserve-69cf8bcfc-msh64\" (UID: \"adb91d40-1c23-456d-afa0-aa6b137e5c0a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64"
Sep 08 13:58:40.546263 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:40.546193    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adb91d40-1c23-456d-afa0-aa6b137e5c0a-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-msh64\" (UID: \"adb91d40-1c23-456d-afa0-aa6b137e5c0a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64"
Sep 08 13:58:40.546391 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:40.546311    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adb91d40-1c23-456d-afa0-aa6b137e5c0a-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-msh64\" (UID: \"adb91d40-1c23-456d-afa0-aa6b137e5c0a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64"
Sep 08 13:58:40.647176 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:40.647126    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adb91d40-1c23-456d-afa0-aa6b137e5c0a-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-msh64\" (UID: \"adb91d40-1c23-456d-afa0-aa6b137e5c0a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64"
Sep 08 13:58:40.647357 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:40.647297    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adb91d40-1c23-456d-afa0-aa6b137e5c0a-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-msh64\" (UID: \"adb91d40-1c23-456d-afa0-aa6b137e5c0a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64"
Sep 08 13:58:40.647405 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:40.647382    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adb91d40-1c23-456d-afa0-aa6b137e5c0a-home\") pod \"roll-v1-kserve-69cf8bcfc-msh64\" (UID: \"adb91d40-1c23-456d-afa0-aa6b137e5c0a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64"
Sep 08 13:58:40.647442 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:40.647412    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adb91d40-1c23-456d-afa0-aa6b137e5c0a-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-msh64\" (UID: \"adb91d40-1c23-456d-afa0-aa6b137e5c0a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64"
Sep 08 13:58:40.647485 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:40.647476    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adb91d40-1c23-456d-afa0-aa6b137e5c0a-dshm\") pod \"roll-v1-kserve-69cf8bcfc-msh64\" (UID: \"adb91d40-1c23-456d-afa0-aa6b137e5c0a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64"
Sep 08 13:58:40.647526 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:40.647509    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhhwv\" (UniqueName: \"kubernetes.io/projected/adb91d40-1c23-456d-afa0-aa6b137e5c0a-kube-api-access-zhhwv\") pod \"roll-v1-kserve-69cf8bcfc-msh64\" (UID: \"adb91d40-1c23-456d-afa0-aa6b137e5c0a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64"
Sep 08 13:58:40.647634 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:40.647586    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adb91d40-1c23-456d-afa0-aa6b137e5c0a-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-msh64\" (UID: \"adb91d40-1c23-456d-afa0-aa6b137e5c0a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64"
Sep 08 13:58:40.647734 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:40.647716    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adb91d40-1c23-456d-afa0-aa6b137e5c0a-home\") pod \"roll-v1-kserve-69cf8bcfc-msh64\" (UID: \"adb91d40-1c23-456d-afa0-aa6b137e5c0a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64"
Sep 08 13:58:40.647879 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:40.647849    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adb91d40-1c23-456d-afa0-aa6b137e5c0a-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-msh64\" (UID: \"adb91d40-1c23-456d-afa0-aa6b137e5c0a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64"
Sep 08 13:58:40.649705 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:40.649673    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adb91d40-1c23-456d-afa0-aa6b137e5c0a-dshm\") pod \"roll-v1-kserve-69cf8bcfc-msh64\" (UID: \"adb91d40-1c23-456d-afa0-aa6b137e5c0a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64"
Sep 08 13:58:40.649963 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:40.649944    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adb91d40-1c23-456d-afa0-aa6b137e5c0a-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-msh64\" (UID: \"adb91d40-1c23-456d-afa0-aa6b137e5c0a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64"
Sep 08 13:58:40.655144 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:40.655124    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhhwv\" (UniqueName: \"kubernetes.io/projected/adb91d40-1c23-456d-afa0-aa6b137e5c0a-kube-api-access-zhhwv\") pod \"roll-v1-kserve-69cf8bcfc-msh64\" (UID: \"adb91d40-1c23-456d-afa0-aa6b137e5c0a\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64"
Sep 08 13:58:40.823769 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:40.823672    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64"
Sep 08 13:58:41.102452 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:41.102414    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64"]
Sep 08 13:58:41.103306 ip-10-0-128-147 kubenswrapper[2558]: W0908 13:58:41.103264    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb91d40_1c23_456d_afa0_aa6b137e5c0a.slice/crio-72edea06e880ecbd9bb1fba03c10189b4beb5ee8270ee806807d788f46c40627 WatchSource:0}: Error finding container 72edea06e880ecbd9bb1fba03c10189b4beb5ee8270ee806807d788f46c40627: Status 404 returned error can't find the container with id 72edea06e880ecbd9bb1fba03c10189b4beb5ee8270ee806807d788f46c40627
Sep 08 13:58:41.529937 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:41.529904    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64" event={"ID":"adb91d40-1c23-456d-afa0-aa6b137e5c0a","Type":"ContainerStarted","Data":"5f3aa0d5e4f3cc60f4dfda7a4b95e37d0370ace42560c87ce546830a3755032d"}
Sep 08 13:58:41.530335 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:41.529953    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64" event={"ID":"adb91d40-1c23-456d-afa0-aa6b137e5c0a","Type":"ContainerStarted","Data":"72edea06e880ecbd9bb1fba03c10189b4beb5ee8270ee806807d788f46c40627"}
Sep 08 13:58:41.549231 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:41.549174    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64" podStartSLOduration=1.549155673 podStartE2EDuration="1.549155673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:58:41.547543304 +0000 UTC m=+2584.999304235" watchObservedRunningTime="2026-09-08 13:58:41.549155673 +0000 UTC m=+2585.000916605"
Sep 08 13:58:41.824056 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:41.823966    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64"
Sep 08 13:58:50.824533 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:50.824491    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64"
Sep 08 13:58:50.837281 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:50.837253    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64"
Sep 08 13:58:51.594581 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:58:51.594550    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64"
Sep 08 13:59:05.149059 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:59:05.148970    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb"]
Sep 08 13:59:05.158796 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:59:05.158755    2558 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 08 13:59:05.158796 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:59:05.158796 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:59:05.158796 ip-10-0-128-147 kubenswrapper[2558]:  > containerID="815a2d5157b2db7a26562e391b344ae7b9bbb9fe23271070736a9d9e0a417651" cmd=["/bin/sleep","15"]
Sep 08 13:59:05.159138 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:59:05.158815    2558 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb" message="" err=<
Sep 08 13:59:05.159138 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:59:05.159138 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:59:05.159138 ip-10-0-128-147 kubenswrapper[2558]:  >
Sep 08 13:59:05.159138 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:59:05.158827    2558 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 08 13:59:05.159138 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 13:59:05.159138 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 13:59:05.159138 ip-10-0-128-147 kubenswrapper[2558]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb" podUID="a4790bad-6605-422b-bdf7-30d69b575c4e" containerName="main" containerID="cri-o://815a2d5157b2db7a26562e391b344ae7b9bbb9fe23271070736a9d9e0a417651"
Sep 08 13:59:05.159138 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:59:05.158865    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb" podUID="a4790bad-6605-422b-bdf7-30d69b575c4e" containerName="main" containerID="cri-o://815a2d5157b2db7a26562e391b344ae7b9bbb9fe23271070736a9d9e0a417651" gracePeriod=60
Sep 08 13:59:05.303860 ip-10-0-128-147 kubenswrapper[2558]: E0908 13:59:05.303820    2558 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4790bad_6605_422b_bdf7_30d69b575c4e.slice/crio-conmon-815a2d5157b2db7a26562e391b344ae7b9bbb9fe23271070736a9d9e0a417651.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4790bad_6605_422b_bdf7_30d69b575c4e.slice/crio-815a2d5157b2db7a26562e391b344ae7b9bbb9fe23271070736a9d9e0a417651.scope\": RecentStats: unable to find data in memory cache]"
Sep 08 13:59:05.305410 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:59:05.305376    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb" podUID="a4790bad-6605-422b-bdf7-30d69b575c4e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": dial tcp 10.132.0.63:8000: connect: connection refused"
Sep 08 13:59:05.650576 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:59:05.650539    2558 generic.go:356] "Generic (PLEG): container finished" podID="a4790bad-6605-422b-bdf7-30d69b575c4e" containerID="815a2d5157b2db7a26562e391b344ae7b9bbb9fe23271070736a9d9e0a417651" exitCode=0
Sep 08 13:59:05.650758 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:59:05.650628    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb" event={"ID":"a4790bad-6605-422b-bdf7-30d69b575c4e","Type":"ContainerDied","Data":"815a2d5157b2db7a26562e391b344ae7b9bbb9fe23271070736a9d9e0a417651"}
Sep 08 13:59:05.911245 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:59:05.911175    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb"
Sep 08 13:59:05.963732 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:59:05.963694    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4790bad-6605-422b-bdf7-30d69b575c4e-dshm\") pod \"a4790bad-6605-422b-bdf7-30d69b575c4e\" (UID: \"a4790bad-6605-422b-bdf7-30d69b575c4e\") "
Sep 08 13:59:05.963915 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:59:05.963757    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sz5vx\" (UniqueName: \"kubernetes.io/projected/a4790bad-6605-422b-bdf7-30d69b575c4e-kube-api-access-sz5vx\") pod \"a4790bad-6605-422b-bdf7-30d69b575c4e\" (UID: \"a4790bad-6605-422b-bdf7-30d69b575c4e\") "
Sep 08 13:59:05.963915 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:59:05.963843    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4790bad-6605-422b-bdf7-30d69b575c4e-tmp-dir\") pod \"a4790bad-6605-422b-bdf7-30d69b575c4e\" (UID: \"a4790bad-6605-422b-bdf7-30d69b575c4e\") "
Sep 08 13:59:05.963915 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:59:05.963870    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4790bad-6605-422b-bdf7-30d69b575c4e-tls-certs\") pod \"a4790bad-6605-422b-bdf7-30d69b575c4e\" (UID: \"a4790bad-6605-422b-bdf7-30d69b575c4e\") "
Sep 08 13:59:05.964176 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:59:05.963992    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4790bad-6605-422b-bdf7-30d69b575c4e-model-cache\") pod \"a4790bad-6605-422b-bdf7-30d69b575c4e\" (UID: \"a4790bad-6605-422b-bdf7-30d69b575c4e\") "
Sep 08 13:59:05.964176 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:59:05.964037    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4790bad-6605-422b-bdf7-30d69b575c4e-home\") pod \"a4790bad-6605-422b-bdf7-30d69b575c4e\" (UID: \"a4790bad-6605-422b-bdf7-30d69b575c4e\") "
Sep 08 13:59:05.964176 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:59:05.964133    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4790bad-6605-422b-bdf7-30d69b575c4e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a4790bad-6605-422b-bdf7-30d69b575c4e" (UID: "a4790bad-6605-422b-bdf7-30d69b575c4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:59:05.964357 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:59:05.964329    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4790bad-6605-422b-bdf7-30d69b575c4e-model-cache" (OuterVolumeSpecName: "model-cache") pod "a4790bad-6605-422b-bdf7-30d69b575c4e" (UID: "a4790bad-6605-422b-bdf7-30d69b575c4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:59:05.964420 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:59:05.964385    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4790bad-6605-422b-bdf7-30d69b575c4e-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:59:05.964531 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:59:05.964511    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4790bad-6605-422b-bdf7-30d69b575c4e-home" (OuterVolumeSpecName: "home") pod "a4790bad-6605-422b-bdf7-30d69b575c4e" (UID: "a4790bad-6605-422b-bdf7-30d69b575c4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:59:05.966133 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:59:05.966100    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4790bad-6605-422b-bdf7-30d69b575c4e-dshm" (OuterVolumeSpecName: "dshm") pod "a4790bad-6605-422b-bdf7-30d69b575c4e" (UID: "a4790bad-6605-422b-bdf7-30d69b575c4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:59:05.966233 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:59:05.966128    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4790bad-6605-422b-bdf7-30d69b575c4e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a4790bad-6605-422b-bdf7-30d69b575c4e" (UID: "a4790bad-6605-422b-bdf7-30d69b575c4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:59:05.966233 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:59:05.966181    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4790bad-6605-422b-bdf7-30d69b575c4e-kube-api-access-sz5vx" (OuterVolumeSpecName: "kube-api-access-sz5vx") pod "a4790bad-6605-422b-bdf7-30d69b575c4e" (UID: "a4790bad-6605-422b-bdf7-30d69b575c4e"). InnerVolumeSpecName "kube-api-access-sz5vx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:59:06.065156 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:59:06.065116    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4790bad-6605-422b-bdf7-30d69b575c4e-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:59:06.065156 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:59:06.065148    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4790bad-6605-422b-bdf7-30d69b575c4e-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:59:06.065156 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:59:06.065157    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4790bad-6605-422b-bdf7-30d69b575c4e-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:59:06.065156 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:59:06.065164    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4790bad-6605-422b-bdf7-30d69b575c4e-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:59:06.065431 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:59:06.065173    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz5vx\" (UniqueName: \"kubernetes.io/projected/a4790bad-6605-422b-bdf7-30d69b575c4e-kube-api-access-sz5vx\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 13:59:06.657281 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:59:06.657250    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb"
Sep 08 13:59:06.657740 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:59:06.657281    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb" event={"ID":"a4790bad-6605-422b-bdf7-30d69b575c4e","Type":"ContainerDied","Data":"4b0693ae1a2d0fe54e2c4337b3ad09915e62649e354d3bf95854ac3e4c1157d8"}
Sep 08 13:59:06.657740 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:59:06.657345    2558 scope.go:117] "RemoveContainer" containerID="815a2d5157b2db7a26562e391b344ae7b9bbb9fe23271070736a9d9e0a417651"
Sep 08 13:59:06.682441 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:59:06.682403    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb"]
Sep 08 13:59:06.688424 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:59:06.688398    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9958c8gb"]
Sep 08 13:59:07.053470 ip-10-0-128-147 kubenswrapper[2558]: I0908 13:59:07.053373    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4790bad-6605-422b-bdf7-30d69b575c4e" path="/var/lib/kubelet/pods/a4790bad-6605-422b-bdf7-30d69b575c4e/volumes"
Sep 08 14:00:17.216997 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:17.216949    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64"]
Sep 08 14:00:23.511776 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:23.511733    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc"]
Sep 08 14:00:23.512305 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:23.512284    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4790bad-6605-422b-bdf7-30d69b575c4e" containerName="main"
Sep 08 14:00:23.512381 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:23.512308    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4790bad-6605-422b-bdf7-30d69b575c4e" containerName="main"
Sep 08 14:00:23.512438 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:23.512420    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4790bad-6605-422b-bdf7-30d69b575c4e" containerName="main"
Sep 08 14:00:23.515648 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:23.515626    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc"
Sep 08 14:00:23.519185 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:23.519162    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"default-dockercfg-xdg9w\""
Sep 08 14:00:23.519346 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:23.519206    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"openshift-service-ca.crt\""
Sep 08 14:00:23.519346 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:23.519225    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v1-kserve-self-signed-certs\""
Sep 08 14:00:23.519346 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:23.519308    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"kube-root-ca.crt\""
Sep 08 14:00:23.526458 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:23.526435    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc"]
Sep 08 14:00:23.546962 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:23.546930    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-home\") pod \"owner-v1-kserve-75579ff5bc-v5shc\" (UID: \"272c57c1-83b2-4b98-8e3d-abd9caf07ee6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc"
Sep 08 14:00:23.547116 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:23.546973    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmnz5\" (UniqueName: \"kubernetes.io/projected/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-kube-api-access-bmnz5\") pod \"owner-v1-kserve-75579ff5bc-v5shc\" (UID: \"272c57c1-83b2-4b98-8e3d-abd9caf07ee6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc"
Sep 08 14:00:23.547116 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:23.547029    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-dshm\") pod \"owner-v1-kserve-75579ff5bc-v5shc\" (UID: \"272c57c1-83b2-4b98-8e3d-abd9caf07ee6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc"
Sep 08 14:00:23.547116 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:23.547054    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-v5shc\" (UID: \"272c57c1-83b2-4b98-8e3d-abd9caf07ee6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc"
Sep 08 14:00:23.547116 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:23.547067    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-v5shc\" (UID: \"272c57c1-83b2-4b98-8e3d-abd9caf07ee6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc"
Sep 08 14:00:23.547259 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:23.547144    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-model-cache\") pod \"owner-v1-kserve-75579ff5bc-v5shc\" (UID: \"272c57c1-83b2-4b98-8e3d-abd9caf07ee6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc"
Sep 08 14:00:23.647682 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:23.647637    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-home\") pod \"owner-v1-kserve-75579ff5bc-v5shc\" (UID: \"272c57c1-83b2-4b98-8e3d-abd9caf07ee6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc"
Sep 08 14:00:23.647682 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:23.647688    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmnz5\" (UniqueName: \"kubernetes.io/projected/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-kube-api-access-bmnz5\") pod \"owner-v1-kserve-75579ff5bc-v5shc\" (UID: \"272c57c1-83b2-4b98-8e3d-abd9caf07ee6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc"
Sep 08 14:00:23.647961 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:23.647746    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-dshm\") pod \"owner-v1-kserve-75579ff5bc-v5shc\" (UID: \"272c57c1-83b2-4b98-8e3d-abd9caf07ee6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc"
Sep 08 14:00:23.647961 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:23.647775    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-v5shc\" (UID: \"272c57c1-83b2-4b98-8e3d-abd9caf07ee6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc"
Sep 08 14:00:23.647961 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:23.647797    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-v5shc\" (UID: \"272c57c1-83b2-4b98-8e3d-abd9caf07ee6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc"
Sep 08 14:00:23.647961 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:23.647853    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-model-cache\") pod \"owner-v1-kserve-75579ff5bc-v5shc\" (UID: \"272c57c1-83b2-4b98-8e3d-abd9caf07ee6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc"
Sep 08 14:00:23.648172 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:23.648074    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-home\") pod \"owner-v1-kserve-75579ff5bc-v5shc\" (UID: \"272c57c1-83b2-4b98-8e3d-abd9caf07ee6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc"
Sep 08 14:00:23.648172 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:23.648155    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-v5shc\" (UID: \"272c57c1-83b2-4b98-8e3d-abd9caf07ee6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc"
Sep 08 14:00:23.648270 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:23.648245    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-model-cache\") pod \"owner-v1-kserve-75579ff5bc-v5shc\" (UID: \"272c57c1-83b2-4b98-8e3d-abd9caf07ee6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc"
Sep 08 14:00:23.650114 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:23.650090    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-dshm\") pod \"owner-v1-kserve-75579ff5bc-v5shc\" (UID: \"272c57c1-83b2-4b98-8e3d-abd9caf07ee6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc"
Sep 08 14:00:23.650447 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:23.650428    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-v5shc\" (UID: \"272c57c1-83b2-4b98-8e3d-abd9caf07ee6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc"
Sep 08 14:00:23.655433 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:23.655406    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmnz5\" (UniqueName: \"kubernetes.io/projected/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-kube-api-access-bmnz5\") pod \"owner-v1-kserve-75579ff5bc-v5shc\" (UID: \"272c57c1-83b2-4b98-8e3d-abd9caf07ee6\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc"
Sep 08 14:00:23.827703 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:23.827612    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc"
Sep 08 14:00:24.110772 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:24.107714    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc"]
Sep 08 14:00:24.111612 ip-10-0-128-147 kubenswrapper[2558]: W0908 14:00:24.111559    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod272c57c1_83b2_4b98_8e3d_abd9caf07ee6.slice/crio-da385aaa9fdfaae465a4091d13157a424eca15f8a5c375f0a491fef04f1e5364 WatchSource:0}: Error finding container da385aaa9fdfaae465a4091d13157a424eca15f8a5c375f0a491fef04f1e5364: Status 404 returned error can't find the container with id da385aaa9fdfaae465a4091d13157a424eca15f8a5c375f0a491fef04f1e5364
Sep 08 14:00:25.038233 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:25.038187    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc" event={"ID":"272c57c1-83b2-4b98-8e3d-abd9caf07ee6","Type":"ContainerStarted","Data":"5ea5e1108d14206aeab4797d17850aec3733f914a3cb951f25c0ae0704bde481"}
Sep 08 14:00:25.038233 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:25.038244    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc" event={"ID":"272c57c1-83b2-4b98-8e3d-abd9caf07ee6","Type":"ContainerStarted","Data":"da385aaa9fdfaae465a4091d13157a424eca15f8a5c375f0a491fef04f1e5364"}
Sep 08 14:00:25.073231 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:25.073178    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc" podStartSLOduration=2.073142242 podStartE2EDuration="2.073142242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:00:25.071097672 +0000 UTC m=+2688.522858617" watchObservedRunningTime="2026-09-08 14:00:25.073142242 +0000 UTC m=+2688.524903192"
Sep 08 14:00:25.828108 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:25.828069    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc"
Sep 08 14:00:28.193163 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:28.193126    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55"]
Sep 08 14:00:28.197547 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:28.197519    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55"
Sep 08 14:00:28.199943 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:28.199916    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v2-kserve-self-signed-certs\""
Sep 08 14:00:28.210100 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:28.210073    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55"]
Sep 08 14:00:28.289875 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:28.289839    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/615f75d1-7978-4144-b414-a5f00bb6e8cc-dshm\") pod \"owner-v2-kserve-5cf8959f95-n5f55\" (UID: \"615f75d1-7978-4144-b414-a5f00bb6e8cc\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55"
Sep 08 14:00:28.290040 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:28.289886    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/615f75d1-7978-4144-b414-a5f00bb6e8cc-model-cache\") pod \"owner-v2-kserve-5cf8959f95-n5f55\" (UID: \"615f75d1-7978-4144-b414-a5f00bb6e8cc\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55"
Sep 08 14:00:28.290040 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:28.289938    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz54w\" (UniqueName: \"kubernetes.io/projected/615f75d1-7978-4144-b414-a5f00bb6e8cc-kube-api-access-gz54w\") pod \"owner-v2-kserve-5cf8959f95-n5f55\" (UID: \"615f75d1-7978-4144-b414-a5f00bb6e8cc\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55"
Sep 08 14:00:28.290040 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:28.289990    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/615f75d1-7978-4144-b414-a5f00bb6e8cc-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-n5f55\" (UID: \"615f75d1-7978-4144-b414-a5f00bb6e8cc\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55"
Sep 08 14:00:28.290185 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:28.290050    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/615f75d1-7978-4144-b414-a5f00bb6e8cc-home\") pod \"owner-v2-kserve-5cf8959f95-n5f55\" (UID: \"615f75d1-7978-4144-b414-a5f00bb6e8cc\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55"
Sep 08 14:00:28.290185 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:28.290085    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/615f75d1-7978-4144-b414-a5f00bb6e8cc-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-n5f55\" (UID: \"615f75d1-7978-4144-b414-a5f00bb6e8cc\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55"
Sep 08 14:00:28.391340 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:28.391296    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz54w\" (UniqueName: \"kubernetes.io/projected/615f75d1-7978-4144-b414-a5f00bb6e8cc-kube-api-access-gz54w\") pod \"owner-v2-kserve-5cf8959f95-n5f55\" (UID: \"615f75d1-7978-4144-b414-a5f00bb6e8cc\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55"
Sep 08 14:00:28.391532 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:28.391352    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/615f75d1-7978-4144-b414-a5f00bb6e8cc-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-n5f55\" (UID: \"615f75d1-7978-4144-b414-a5f00bb6e8cc\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55"
Sep 08 14:00:28.391532 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:28.391395    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/615f75d1-7978-4144-b414-a5f00bb6e8cc-home\") pod \"owner-v2-kserve-5cf8959f95-n5f55\" (UID: \"615f75d1-7978-4144-b414-a5f00bb6e8cc\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55"
Sep 08 14:00:28.391532 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:28.391430    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/615f75d1-7978-4144-b414-a5f00bb6e8cc-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-n5f55\" (UID: \"615f75d1-7978-4144-b414-a5f00bb6e8cc\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55"
Sep 08 14:00:28.391532 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:28.391478    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/615f75d1-7978-4144-b414-a5f00bb6e8cc-dshm\") pod \"owner-v2-kserve-5cf8959f95-n5f55\" (UID: \"615f75d1-7978-4144-b414-a5f00bb6e8cc\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55"
Sep 08 14:00:28.391532 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:28.391510    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/615f75d1-7978-4144-b414-a5f00bb6e8cc-model-cache\") pod \"owner-v2-kserve-5cf8959f95-n5f55\" (UID: \"615f75d1-7978-4144-b414-a5f00bb6e8cc\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55"
Sep 08 14:00:28.391920 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:28.391895    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/615f75d1-7978-4144-b414-a5f00bb6e8cc-home\") pod \"owner-v2-kserve-5cf8959f95-n5f55\" (UID: \"615f75d1-7978-4144-b414-a5f00bb6e8cc\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55"
Sep 08 14:00:28.391969 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:28.391927    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/615f75d1-7978-4144-b414-a5f00bb6e8cc-model-cache\") pod \"owner-v2-kserve-5cf8959f95-n5f55\" (UID: \"615f75d1-7978-4144-b414-a5f00bb6e8cc\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55"
Sep 08 14:00:28.391969 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:28.391953    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/615f75d1-7978-4144-b414-a5f00bb6e8cc-tmp-dir\") pod \"owner-v2-kserve-5cf8959f95-n5f55\" (UID: \"615f75d1-7978-4144-b414-a5f00bb6e8cc\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55"
Sep 08 14:00:28.393898 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:28.393877    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/615f75d1-7978-4144-b414-a5f00bb6e8cc-dshm\") pod \"owner-v2-kserve-5cf8959f95-n5f55\" (UID: \"615f75d1-7978-4144-b414-a5f00bb6e8cc\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55"
Sep 08 14:00:28.393992 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:28.393977    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/615f75d1-7978-4144-b414-a5f00bb6e8cc-tls-certs\") pod \"owner-v2-kserve-5cf8959f95-n5f55\" (UID: \"615f75d1-7978-4144-b414-a5f00bb6e8cc\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55"
Sep 08 14:00:28.399407 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:28.399382    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz54w\" (UniqueName: \"kubernetes.io/projected/615f75d1-7978-4144-b414-a5f00bb6e8cc-kube-api-access-gz54w\") pod \"owner-v2-kserve-5cf8959f95-n5f55\" (UID: \"615f75d1-7978-4144-b414-a5f00bb6e8cc\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55"
Sep 08 14:00:28.508873 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:28.508771    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55"
Sep 08 14:00:28.784948 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:28.784814    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55"]
Sep 08 14:00:28.787561 ip-10-0-128-147 kubenswrapper[2558]: W0908 14:00:28.787518    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod615f75d1_7978_4144_b414_a5f00bb6e8cc.slice/crio-d06b70325f9995f07c667f49044157f1f9f295bd55a1ece56254d0a751e40784 WatchSource:0}: Error finding container d06b70325f9995f07c667f49044157f1f9f295bd55a1ece56254d0a751e40784: Status 404 returned error can't find the container with id d06b70325f9995f07c667f49044157f1f9f295bd55a1ece56254d0a751e40784
Sep 08 14:00:29.060715 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:29.060626    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55" event={"ID":"615f75d1-7978-4144-b414-a5f00bb6e8cc","Type":"ContainerStarted","Data":"f4774eb590df0629016a868eafc4d4a4b29d74f235c5118defed6aee39c5a493"}
Sep 08 14:00:29.060715 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:29.060698    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55" event={"ID":"615f75d1-7978-4144-b414-a5f00bb6e8cc","Type":"ContainerStarted","Data":"d06b70325f9995f07c667f49044157f1f9f295bd55a1ece56254d0a751e40784"}
Sep 08 14:00:29.084844 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:29.084784    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55" podStartSLOduration=1.084765239 podStartE2EDuration="1.084765239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:00:29.082474503 +0000 UTC m=+2692.534235447" watchObservedRunningTime="2026-09-08 14:00:29.084765239 +0000 UTC m=+2692.536526192"
Sep 08 14:00:29.509010 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:29.508974    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55"
Sep 08 14:00:32.236629 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:32.236572    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64" podUID="adb91d40-1c23-456d-afa0-aa6b137e5c0a" containerName="main" containerID="cri-o://5f3aa0d5e4f3cc60f4dfda7a4b95e37d0370ace42560c87ce546830a3755032d" gracePeriod=45
Sep 08 14:00:32.487858 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:32.487801    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64"
Sep 08 14:00:32.530905 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:32.530876    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhhwv\" (UniqueName: \"kubernetes.io/projected/adb91d40-1c23-456d-afa0-aa6b137e5c0a-kube-api-access-zhhwv\") pod \"adb91d40-1c23-456d-afa0-aa6b137e5c0a\" (UID: \"adb91d40-1c23-456d-afa0-aa6b137e5c0a\") "
Sep 08 14:00:32.531055 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:32.530918    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adb91d40-1c23-456d-afa0-aa6b137e5c0a-dshm\") pod \"adb91d40-1c23-456d-afa0-aa6b137e5c0a\" (UID: \"adb91d40-1c23-456d-afa0-aa6b137e5c0a\") "
Sep 08 14:00:32.531055 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:32.530936    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adb91d40-1c23-456d-afa0-aa6b137e5c0a-tls-certs\") pod \"adb91d40-1c23-456d-afa0-aa6b137e5c0a\" (UID: \"adb91d40-1c23-456d-afa0-aa6b137e5c0a\") "
Sep 08 14:00:32.531055 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:32.530981    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adb91d40-1c23-456d-afa0-aa6b137e5c0a-home\") pod \"adb91d40-1c23-456d-afa0-aa6b137e5c0a\" (UID: \"adb91d40-1c23-456d-afa0-aa6b137e5c0a\") "
Sep 08 14:00:32.531055 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:32.531000    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adb91d40-1c23-456d-afa0-aa6b137e5c0a-model-cache\") pod \"adb91d40-1c23-456d-afa0-aa6b137e5c0a\" (UID: \"adb91d40-1c23-456d-afa0-aa6b137e5c0a\") "
Sep 08 14:00:32.531270 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:32.531066    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adb91d40-1c23-456d-afa0-aa6b137e5c0a-tmp-dir\") pod \"adb91d40-1c23-456d-afa0-aa6b137e5c0a\" (UID: \"adb91d40-1c23-456d-afa0-aa6b137e5c0a\") "
Sep 08 14:00:32.531325 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:32.531275    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb91d40-1c23-456d-afa0-aa6b137e5c0a-home" (OuterVolumeSpecName: "home") pod "adb91d40-1c23-456d-afa0-aa6b137e5c0a" (UID: "adb91d40-1c23-456d-afa0-aa6b137e5c0a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:00:32.531390 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:32.531321    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb91d40-1c23-456d-afa0-aa6b137e5c0a-model-cache" (OuterVolumeSpecName: "model-cache") pod "adb91d40-1c23-456d-afa0-aa6b137e5c0a" (UID: "adb91d40-1c23-456d-afa0-aa6b137e5c0a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:00:32.531543 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:32.531515    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb91d40-1c23-456d-afa0-aa6b137e5c0a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "adb91d40-1c23-456d-afa0-aa6b137e5c0a" (UID: "adb91d40-1c23-456d-afa0-aa6b137e5c0a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:00:32.533180 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:32.533158    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adb91d40-1c23-456d-afa0-aa6b137e5c0a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "adb91d40-1c23-456d-afa0-aa6b137e5c0a" (UID: "adb91d40-1c23-456d-afa0-aa6b137e5c0a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:00:32.533276 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:32.533252    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb91d40-1c23-456d-afa0-aa6b137e5c0a-kube-api-access-zhhwv" (OuterVolumeSpecName: "kube-api-access-zhhwv") pod "adb91d40-1c23-456d-afa0-aa6b137e5c0a" (UID: "adb91d40-1c23-456d-afa0-aa6b137e5c0a"). InnerVolumeSpecName "kube-api-access-zhhwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:00:32.533439 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:32.533425    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb91d40-1c23-456d-afa0-aa6b137e5c0a-dshm" (OuterVolumeSpecName: "dshm") pod "adb91d40-1c23-456d-afa0-aa6b137e5c0a" (UID: "adb91d40-1c23-456d-afa0-aa6b137e5c0a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:00:32.632427 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:32.632381    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhhwv\" (UniqueName: \"kubernetes.io/projected/adb91d40-1c23-456d-afa0-aa6b137e5c0a-kube-api-access-zhhwv\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:00:32.632427 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:32.632419    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adb91d40-1c23-456d-afa0-aa6b137e5c0a-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:00:32.632427 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:32.632428    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adb91d40-1c23-456d-afa0-aa6b137e5c0a-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:00:32.632427 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:32.632437    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adb91d40-1c23-456d-afa0-aa6b137e5c0a-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:00:32.632722 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:32.632445    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adb91d40-1c23-456d-afa0-aa6b137e5c0a-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:00:32.632722 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:32.632452    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adb91d40-1c23-456d-afa0-aa6b137e5c0a-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:00:33.080825 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:33.080794    2558 generic.go:356] "Generic (PLEG): container finished" podID="adb91d40-1c23-456d-afa0-aa6b137e5c0a" containerID="5f3aa0d5e4f3cc60f4dfda7a4b95e37d0370ace42560c87ce546830a3755032d" exitCode=0
Sep 08 14:00:33.080987 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:33.080833    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64" event={"ID":"adb91d40-1c23-456d-afa0-aa6b137e5c0a","Type":"ContainerDied","Data":"5f3aa0d5e4f3cc60f4dfda7a4b95e37d0370ace42560c87ce546830a3755032d"}
Sep 08 14:00:33.080987 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:33.080846    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64"
Sep 08 14:00:33.080987 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:33.080884    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64" event={"ID":"adb91d40-1c23-456d-afa0-aa6b137e5c0a","Type":"ContainerDied","Data":"72edea06e880ecbd9bb1fba03c10189b4beb5ee8270ee806807d788f46c40627"}
Sep 08 14:00:33.080987 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:33.080915    2558 scope.go:117] "RemoveContainer" containerID="5f3aa0d5e4f3cc60f4dfda7a4b95e37d0370ace42560c87ce546830a3755032d"
Sep 08 14:00:33.089781 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:33.089765    2558 scope.go:117] "RemoveContainer" containerID="5f3aa0d5e4f3cc60f4dfda7a4b95e37d0370ace42560c87ce546830a3755032d"
Sep 08 14:00:33.090029 ip-10-0-128-147 kubenswrapper[2558]: E0908 14:00:33.090008    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f3aa0d5e4f3cc60f4dfda7a4b95e37d0370ace42560c87ce546830a3755032d\": container with ID starting with 5f3aa0d5e4f3cc60f4dfda7a4b95e37d0370ace42560c87ce546830a3755032d not found: ID does not exist" containerID="5f3aa0d5e4f3cc60f4dfda7a4b95e37d0370ace42560c87ce546830a3755032d"
Sep 08 14:00:33.090079 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:33.090037    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f3aa0d5e4f3cc60f4dfda7a4b95e37d0370ace42560c87ce546830a3755032d"} err="failed to get container status \"5f3aa0d5e4f3cc60f4dfda7a4b95e37d0370ace42560c87ce546830a3755032d\": rpc error: code = NotFound desc = could not find container \"5f3aa0d5e4f3cc60f4dfda7a4b95e37d0370ace42560c87ce546830a3755032d\": container with ID starting with 5f3aa0d5e4f3cc60f4dfda7a4b95e37d0370ace42560c87ce546830a3755032d not found: ID does not exist"
Sep 08 14:00:33.097243 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:33.097222    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64"]
Sep 08 14:00:33.101201 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:33.101180    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-msh64"]
Sep 08 14:00:33.828761 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:33.828712    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc"
Sep 08 14:00:33.841534 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:33.841508    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc"
Sep 08 14:00:34.097334 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:34.097258    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc"
Sep 08 14:00:35.054107 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:35.054073    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adb91d40-1c23-456d-afa0-aa6b137e5c0a" path="/var/lib/kubelet/pods/adb91d40-1c23-456d-afa0-aa6b137e5c0a/volumes"
Sep 08 14:00:38.508985 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:38.508946    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55"
Sep 08 14:00:38.521425 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:38.521396    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55"
Sep 08 14:00:39.122026 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:00:39.121997    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55"
Sep 08 14:01:06.368905 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:06.368854    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc"]
Sep 08 14:01:21.390028 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:21.389983    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc" podUID="272c57c1-83b2-4b98-8e3d-abd9caf07ee6" containerName="main" containerID="cri-o://5ea5e1108d14206aeab4797d17850aec3733f914a3cb951f25c0ae0704bde481" gracePeriod=45
Sep 08 14:01:21.645167 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:21.645102    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc"
Sep 08 14:01:21.666172 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:21.666141    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-tls-certs\") pod \"272c57c1-83b2-4b98-8e3d-abd9caf07ee6\" (UID: \"272c57c1-83b2-4b98-8e3d-abd9caf07ee6\") "
Sep 08 14:01:21.666368 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:21.666217    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-model-cache\") pod \"272c57c1-83b2-4b98-8e3d-abd9caf07ee6\" (UID: \"272c57c1-83b2-4b98-8e3d-abd9caf07ee6\") "
Sep 08 14:01:21.666368 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:21.666258    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-dshm\") pod \"272c57c1-83b2-4b98-8e3d-abd9caf07ee6\" (UID: \"272c57c1-83b2-4b98-8e3d-abd9caf07ee6\") "
Sep 08 14:01:21.666368 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:21.666290    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-home\") pod \"272c57c1-83b2-4b98-8e3d-abd9caf07ee6\" (UID: \"272c57c1-83b2-4b98-8e3d-abd9caf07ee6\") "
Sep 08 14:01:21.666368 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:21.666319    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmnz5\" (UniqueName: \"kubernetes.io/projected/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-kube-api-access-bmnz5\") pod \"272c57c1-83b2-4b98-8e3d-abd9caf07ee6\" (UID: \"272c57c1-83b2-4b98-8e3d-abd9caf07ee6\") "
Sep 08 14:01:21.666574 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:21.666383    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-tmp-dir\") pod \"272c57c1-83b2-4b98-8e3d-abd9caf07ee6\" (UID: \"272c57c1-83b2-4b98-8e3d-abd9caf07ee6\") "
Sep 08 14:01:21.666574 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:21.666513    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-model-cache" (OuterVolumeSpecName: "model-cache") pod "272c57c1-83b2-4b98-8e3d-abd9caf07ee6" (UID: "272c57c1-83b2-4b98-8e3d-abd9caf07ee6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:01:21.666820 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:21.666715    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:01:21.666820 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:21.666791    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "272c57c1-83b2-4b98-8e3d-abd9caf07ee6" (UID: "272c57c1-83b2-4b98-8e3d-abd9caf07ee6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:01:21.666945 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:21.666913    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-home" (OuterVolumeSpecName: "home") pod "272c57c1-83b2-4b98-8e3d-abd9caf07ee6" (UID: "272c57c1-83b2-4b98-8e3d-abd9caf07ee6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:01:21.669037 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:21.669007    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-dshm" (OuterVolumeSpecName: "dshm") pod "272c57c1-83b2-4b98-8e3d-abd9caf07ee6" (UID: "272c57c1-83b2-4b98-8e3d-abd9caf07ee6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:01:21.669187 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:21.669103    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-kube-api-access-bmnz5" (OuterVolumeSpecName: "kube-api-access-bmnz5") pod "272c57c1-83b2-4b98-8e3d-abd9caf07ee6" (UID: "272c57c1-83b2-4b98-8e3d-abd9caf07ee6"). InnerVolumeSpecName "kube-api-access-bmnz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:01:21.669187 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:21.669111    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "272c57c1-83b2-4b98-8e3d-abd9caf07ee6" (UID: "272c57c1-83b2-4b98-8e3d-abd9caf07ee6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:01:21.767733 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:21.767694    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:01:21.767733 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:21.767724    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:01:21.767733 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:21.767736    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmnz5\" (UniqueName: \"kubernetes.io/projected/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-kube-api-access-bmnz5\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:01:21.767978 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:21.767755    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:01:21.767978 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:21.767764    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/272c57c1-83b2-4b98-8e3d-abd9caf07ee6-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:01:22.322438 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:22.322392    2558 generic.go:356] "Generic (PLEG): container finished" podID="272c57c1-83b2-4b98-8e3d-abd9caf07ee6" containerID="5ea5e1108d14206aeab4797d17850aec3733f914a3cb951f25c0ae0704bde481" exitCode=0
Sep 08 14:01:22.322438 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:22.322436    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc"
Sep 08 14:01:22.322693 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:22.322439    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc" event={"ID":"272c57c1-83b2-4b98-8e3d-abd9caf07ee6","Type":"ContainerDied","Data":"5ea5e1108d14206aeab4797d17850aec3733f914a3cb951f25c0ae0704bde481"}
Sep 08 14:01:22.322693 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:22.322549    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc" event={"ID":"272c57c1-83b2-4b98-8e3d-abd9caf07ee6","Type":"ContainerDied","Data":"da385aaa9fdfaae465a4091d13157a424eca15f8a5c375f0a491fef04f1e5364"}
Sep 08 14:01:22.322693 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:22.322572    2558 scope.go:117] "RemoveContainer" containerID="5ea5e1108d14206aeab4797d17850aec3733f914a3cb951f25c0ae0704bde481"
Sep 08 14:01:22.333145 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:22.333123    2558 scope.go:117] "RemoveContainer" containerID="5ea5e1108d14206aeab4797d17850aec3733f914a3cb951f25c0ae0704bde481"
Sep 08 14:01:22.333556 ip-10-0-128-147 kubenswrapper[2558]: E0908 14:01:22.333537    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ea5e1108d14206aeab4797d17850aec3733f914a3cb951f25c0ae0704bde481\": container with ID starting with 5ea5e1108d14206aeab4797d17850aec3733f914a3cb951f25c0ae0704bde481 not found: ID does not exist" containerID="5ea5e1108d14206aeab4797d17850aec3733f914a3cb951f25c0ae0704bde481"
Sep 08 14:01:22.333656 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:22.333567    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ea5e1108d14206aeab4797d17850aec3733f914a3cb951f25c0ae0704bde481"} err="failed to get container status \"5ea5e1108d14206aeab4797d17850aec3733f914a3cb951f25c0ae0704bde481\": rpc error: code = NotFound desc = could not find container \"5ea5e1108d14206aeab4797d17850aec3733f914a3cb951f25c0ae0704bde481\": container with ID starting with 5ea5e1108d14206aeab4797d17850aec3733f914a3cb951f25c0ae0704bde481 not found: ID does not exist"
Sep 08 14:01:22.346293 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:22.346264    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc"]
Sep 08 14:01:22.351319 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:22.351292    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-v5shc"]
Sep 08 14:01:23.054027 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:23.053989    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="272c57c1-83b2-4b98-8e3d-abd9caf07ee6" path="/var/lib/kubelet/pods/272c57c1-83b2-4b98-8e3d-abd9caf07ee6/volumes"
Sep 08 14:01:32.516385 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:32.516342    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55"]
Sep 08 14:01:47.535987 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:47.535940    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55" podUID="615f75d1-7978-4144-b414-a5f00bb6e8cc" containerName="main" containerID="cri-o://f4774eb590df0629016a868eafc4d4a4b29d74f235c5118defed6aee39c5a493" gracePeriod=45
Sep 08 14:01:47.786893 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:47.786835    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55"
Sep 08 14:01:47.883726 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:47.883684    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/615f75d1-7978-4144-b414-a5f00bb6e8cc-tmp-dir\") pod \"615f75d1-7978-4144-b414-a5f00bb6e8cc\" (UID: \"615f75d1-7978-4144-b414-a5f00bb6e8cc\") "
Sep 08 14:01:47.883726 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:47.883726    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/615f75d1-7978-4144-b414-a5f00bb6e8cc-tls-certs\") pod \"615f75d1-7978-4144-b414-a5f00bb6e8cc\" (UID: \"615f75d1-7978-4144-b414-a5f00bb6e8cc\") "
Sep 08 14:01:47.883972 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:47.883762    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/615f75d1-7978-4144-b414-a5f00bb6e8cc-home\") pod \"615f75d1-7978-4144-b414-a5f00bb6e8cc\" (UID: \"615f75d1-7978-4144-b414-a5f00bb6e8cc\") "
Sep 08 14:01:47.883972 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:47.883794    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/615f75d1-7978-4144-b414-a5f00bb6e8cc-model-cache\") pod \"615f75d1-7978-4144-b414-a5f00bb6e8cc\" (UID: \"615f75d1-7978-4144-b414-a5f00bb6e8cc\") "
Sep 08 14:01:47.883972 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:47.883845    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gz54w\" (UniqueName: \"kubernetes.io/projected/615f75d1-7978-4144-b414-a5f00bb6e8cc-kube-api-access-gz54w\") pod \"615f75d1-7978-4144-b414-a5f00bb6e8cc\" (UID: \"615f75d1-7978-4144-b414-a5f00bb6e8cc\") "
Sep 08 14:01:47.883972 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:47.883885    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/615f75d1-7978-4144-b414-a5f00bb6e8cc-dshm\") pod \"615f75d1-7978-4144-b414-a5f00bb6e8cc\" (UID: \"615f75d1-7978-4144-b414-a5f00bb6e8cc\") "
Sep 08 14:01:47.884184 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:47.884020    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615f75d1-7978-4144-b414-a5f00bb6e8cc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "615f75d1-7978-4144-b414-a5f00bb6e8cc" (UID: "615f75d1-7978-4144-b414-a5f00bb6e8cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:01:47.884184 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:47.884077    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615f75d1-7978-4144-b414-a5f00bb6e8cc-home" (OuterVolumeSpecName: "home") pod "615f75d1-7978-4144-b414-a5f00bb6e8cc" (UID: "615f75d1-7978-4144-b414-a5f00bb6e8cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:01:47.884184 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:47.884101    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615f75d1-7978-4144-b414-a5f00bb6e8cc-model-cache" (OuterVolumeSpecName: "model-cache") pod "615f75d1-7978-4144-b414-a5f00bb6e8cc" (UID: "615f75d1-7978-4144-b414-a5f00bb6e8cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:01:47.884331 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:47.884236    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/615f75d1-7978-4144-b414-a5f00bb6e8cc-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:01:47.884331 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:47.884257    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/615f75d1-7978-4144-b414-a5f00bb6e8cc-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:01:47.884331 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:47.884270    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/615f75d1-7978-4144-b414-a5f00bb6e8cc-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:01:47.885944 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:47.885917    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/615f75d1-7978-4144-b414-a5f00bb6e8cc-dshm" (OuterVolumeSpecName: "dshm") pod "615f75d1-7978-4144-b414-a5f00bb6e8cc" (UID: "615f75d1-7978-4144-b414-a5f00bb6e8cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:01:47.885944 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:47.885933    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/615f75d1-7978-4144-b414-a5f00bb6e8cc-kube-api-access-gz54w" (OuterVolumeSpecName: "kube-api-access-gz54w") pod "615f75d1-7978-4144-b414-a5f00bb6e8cc" (UID: "615f75d1-7978-4144-b414-a5f00bb6e8cc"). InnerVolumeSpecName "kube-api-access-gz54w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:01:47.886113 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:47.885923    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/615f75d1-7978-4144-b414-a5f00bb6e8cc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "615f75d1-7978-4144-b414-a5f00bb6e8cc" (UID: "615f75d1-7978-4144-b414-a5f00bb6e8cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:01:47.985179 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:47.985152    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz54w\" (UniqueName: \"kubernetes.io/projected/615f75d1-7978-4144-b414-a5f00bb6e8cc-kube-api-access-gz54w\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:01:47.985179 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:47.985175    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/615f75d1-7978-4144-b414-a5f00bb6e8cc-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:01:47.985179 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:47.985185    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/615f75d1-7978-4144-b414-a5f00bb6e8cc-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:01:48.009032 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.009003    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn"]
Sep 08 14:01:48.009390 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.009378    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="272c57c1-83b2-4b98-8e3d-abd9caf07ee6" containerName="main"
Sep 08 14:01:48.009434 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.009392    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="272c57c1-83b2-4b98-8e3d-abd9caf07ee6" containerName="main"
Sep 08 14:01:48.009434 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.009401    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="615f75d1-7978-4144-b414-a5f00bb6e8cc" containerName="main"
Sep 08 14:01:48.009434 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.009409    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="615f75d1-7978-4144-b414-a5f00bb6e8cc" containerName="main"
Sep 08 14:01:48.009434 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.009418    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adb91d40-1c23-456d-afa0-aa6b137e5c0a" containerName="main"
Sep 08 14:01:48.009434 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.009424    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="adb91d40-1c23-456d-afa0-aa6b137e5c0a" containerName="main"
Sep 08 14:01:48.009620 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.009503    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="adb91d40-1c23-456d-afa0-aa6b137e5c0a" containerName="main"
Sep 08 14:01:48.009620 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.009511    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="272c57c1-83b2-4b98-8e3d-abd9caf07ee6" containerName="main"
Sep 08 14:01:48.009620 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.009526    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="615f75d1-7978-4144-b414-a5f00bb6e8cc" containerName="main"
Sep 08 14:01:48.012687 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.012669    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn"
Sep 08 14:01:48.015147 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.015125    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-ssptd\""
Sep 08 14:01:48.015258 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.015178    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Sep 08 14:01:48.015465 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.015443    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 08 14:01:48.015465 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.015461    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 08 14:01:48.023863 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.023845    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn"]
Sep 08 14:01:48.086351 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.086276    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c969b297-3352-4bc2-8a5e-1a464927c074-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn\" (UID: \"c969b297-3352-4bc2-8a5e-1a464927c074\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn"
Sep 08 14:01:48.086351 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.086311    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c969b297-3352-4bc2-8a5e-1a464927c074-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn\" (UID: \"c969b297-3352-4bc2-8a5e-1a464927c074\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn"
Sep 08 14:01:48.086351 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.086339    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw62m\" (UniqueName: \"kubernetes.io/projected/c969b297-3352-4bc2-8a5e-1a464927c074-kube-api-access-lw62m\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn\" (UID: \"c969b297-3352-4bc2-8a5e-1a464927c074\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn"
Sep 08 14:01:48.086557 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.086452    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c969b297-3352-4bc2-8a5e-1a464927c074-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn\" (UID: \"c969b297-3352-4bc2-8a5e-1a464927c074\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn"
Sep 08 14:01:48.086557 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.086487    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c969b297-3352-4bc2-8a5e-1a464927c074-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn\" (UID: \"c969b297-3352-4bc2-8a5e-1a464927c074\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn"
Sep 08 14:01:48.086557 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.086513    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c969b297-3352-4bc2-8a5e-1a464927c074-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn\" (UID: \"c969b297-3352-4bc2-8a5e-1a464927c074\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn"
Sep 08 14:01:48.187654 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.187618    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c969b297-3352-4bc2-8a5e-1a464927c074-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn\" (UID: \"c969b297-3352-4bc2-8a5e-1a464927c074\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn"
Sep 08 14:01:48.187837 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.187702    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c969b297-3352-4bc2-8a5e-1a464927c074-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn\" (UID: \"c969b297-3352-4bc2-8a5e-1a464927c074\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn"
Sep 08 14:01:48.187837 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.187732    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c969b297-3352-4bc2-8a5e-1a464927c074-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn\" (UID: \"c969b297-3352-4bc2-8a5e-1a464927c074\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn"
Sep 08 14:01:48.187837 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.187773    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw62m\" (UniqueName: \"kubernetes.io/projected/c969b297-3352-4bc2-8a5e-1a464927c074-kube-api-access-lw62m\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn\" (UID: \"c969b297-3352-4bc2-8a5e-1a464927c074\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn"
Sep 08 14:01:48.187996 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.187841    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c969b297-3352-4bc2-8a5e-1a464927c074-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn\" (UID: \"c969b297-3352-4bc2-8a5e-1a464927c074\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn"
Sep 08 14:01:48.187996 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.187882    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c969b297-3352-4bc2-8a5e-1a464927c074-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn\" (UID: \"c969b297-3352-4bc2-8a5e-1a464927c074\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn"
Sep 08 14:01:48.188111 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.188077    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c969b297-3352-4bc2-8a5e-1a464927c074-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn\" (UID: \"c969b297-3352-4bc2-8a5e-1a464927c074\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn"
Sep 08 14:01:48.188111 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.188095    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c969b297-3352-4bc2-8a5e-1a464927c074-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn\" (UID: \"c969b297-3352-4bc2-8a5e-1a464927c074\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn"
Sep 08 14:01:48.188221 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.188191    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c969b297-3352-4bc2-8a5e-1a464927c074-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn\" (UID: \"c969b297-3352-4bc2-8a5e-1a464927c074\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn"
Sep 08 14:01:48.190036 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.190019    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c969b297-3352-4bc2-8a5e-1a464927c074-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn\" (UID: \"c969b297-3352-4bc2-8a5e-1a464927c074\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn"
Sep 08 14:01:48.190236 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.190219    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c969b297-3352-4bc2-8a5e-1a464927c074-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn\" (UID: \"c969b297-3352-4bc2-8a5e-1a464927c074\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn"
Sep 08 14:01:48.195694 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.195671    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw62m\" (UniqueName: \"kubernetes.io/projected/c969b297-3352-4bc2-8a5e-1a464927c074-kube-api-access-lw62m\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn\" (UID: \"c969b297-3352-4bc2-8a5e-1a464927c074\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn"
Sep 08 14:01:48.324482 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.324447    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn"
Sep 08 14:01:48.447791 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.447754    2558 generic.go:356] "Generic (PLEG): container finished" podID="615f75d1-7978-4144-b414-a5f00bb6e8cc" containerID="f4774eb590df0629016a868eafc4d4a4b29d74f235c5118defed6aee39c5a493" exitCode=0
Sep 08 14:01:48.447968 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.447801    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55"
Sep 08 14:01:48.447968 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.447808    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55" event={"ID":"615f75d1-7978-4144-b414-a5f00bb6e8cc","Type":"ContainerDied","Data":"f4774eb590df0629016a868eafc4d4a4b29d74f235c5118defed6aee39c5a493"}
Sep 08 14:01:48.447968 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.447857    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55" event={"ID":"615f75d1-7978-4144-b414-a5f00bb6e8cc","Type":"ContainerDied","Data":"d06b70325f9995f07c667f49044157f1f9f295bd55a1ece56254d0a751e40784"}
Sep 08 14:01:48.447968 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.447879    2558 scope.go:117] "RemoveContainer" containerID="f4774eb590df0629016a868eafc4d4a4b29d74f235c5118defed6aee39c5a493"
Sep 08 14:01:48.465142 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.465119    2558 scope.go:117] "RemoveContainer" containerID="f4774eb590df0629016a868eafc4d4a4b29d74f235c5118defed6aee39c5a493"
Sep 08 14:01:48.465513 ip-10-0-128-147 kubenswrapper[2558]: E0908 14:01:48.465491    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4774eb590df0629016a868eafc4d4a4b29d74f235c5118defed6aee39c5a493\": container with ID starting with f4774eb590df0629016a868eafc4d4a4b29d74f235c5118defed6aee39c5a493 not found: ID does not exist" containerID="f4774eb590df0629016a868eafc4d4a4b29d74f235c5118defed6aee39c5a493"
Sep 08 14:01:48.465587 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.465521    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4774eb590df0629016a868eafc4d4a4b29d74f235c5118defed6aee39c5a493"} err="failed to get container status \"f4774eb590df0629016a868eafc4d4a4b29d74f235c5118defed6aee39c5a493\": rpc error: code = NotFound desc = could not find container \"f4774eb590df0629016a868eafc4d4a4b29d74f235c5118defed6aee39c5a493\": container with ID starting with f4774eb590df0629016a868eafc4d4a4b29d74f235c5118defed6aee39c5a493 not found: ID does not exist"
Sep 08 14:01:48.479109 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.479081    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55"]
Sep 08 14:01:48.483128 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.483097    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v2-kserve-5cf8959f95-n5f55"]
Sep 08 14:01:48.603471 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:48.603445    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn"]
Sep 08 14:01:48.606948 ip-10-0-128-147 kubenswrapper[2558]: W0908 14:01:48.606921    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc969b297_3352_4bc2_8a5e_1a464927c074.slice/crio-ee4a9dabaee99f03641f62686233abc48c41fea303130a1b3bcbcfa8b975df77 WatchSource:0}: Error finding container ee4a9dabaee99f03641f62686233abc48c41fea303130a1b3bcbcfa8b975df77: Status 404 returned error can't find the container with id ee4a9dabaee99f03641f62686233abc48c41fea303130a1b3bcbcfa8b975df77
Sep 08 14:01:49.053493 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:49.053408    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="615f75d1-7978-4144-b414-a5f00bb6e8cc" path="/var/lib/kubelet/pods/615f75d1-7978-4144-b414-a5f00bb6e8cc/volumes"
Sep 08 14:01:49.454004 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:49.453968    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn" event={"ID":"c969b297-3352-4bc2-8a5e-1a464927c074","Type":"ContainerStarted","Data":"b9ffbd60d01e7bbf0fa408a46241eb2f034cff4c6423092cedc00c00c7159693"}
Sep 08 14:01:49.454202 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:49.454030    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn" event={"ID":"c969b297-3352-4bc2-8a5e-1a464927c074","Type":"ContainerStarted","Data":"ee4a9dabaee99f03641f62686233abc48c41fea303130a1b3bcbcfa8b975df77"}
Sep 08 14:01:49.479761 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:49.479711    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn" podStartSLOduration=2.479694871 podStartE2EDuration="2.479694871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:01:49.477588035 +0000 UTC m=+2772.929348966" watchObservedRunningTime="2026-09-08 14:01:49.479694871 +0000 UTC m=+2772.931455796"
Sep 08 14:01:50.325062 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:50.325020    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn"
Sep 08 14:01:58.325166 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:58.325126    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn"
Sep 08 14:01:58.338031 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:58.337999    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn"
Sep 08 14:01:58.510580 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:01:58.510544    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn"
Sep 08 14:02:37.320864 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:37.320821    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn"]
Sep 08 14:02:37.329871 ip-10-0-128-147 kubenswrapper[2558]: E0908 14:02:37.329837    2558 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 08 14:02:37.329871 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 14:02:37.329871 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 14:02:37.329871 ip-10-0-128-147 kubenswrapper[2558]:  > containerID="b9ffbd60d01e7bbf0fa408a46241eb2f034cff4c6423092cedc00c00c7159693" cmd=["/bin/sleep","15"]
Sep 08 14:02:37.330127 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:37.329887    2558 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn" message="" err=<
Sep 08 14:02:37.330127 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 14:02:37.330127 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 14:02:37.330127 ip-10-0-128-147 kubenswrapper[2558]:  >
Sep 08 14:02:37.330127 ip-10-0-128-147 kubenswrapper[2558]: E0908 14:02:37.329899    2558 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 08 14:02:37.330127 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 14:02:37.330127 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 14:02:37.330127 ip-10-0-128-147 kubenswrapper[2558]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn" podUID="c969b297-3352-4bc2-8a5e-1a464927c074" containerName="main" containerID="cri-o://b9ffbd60d01e7bbf0fa408a46241eb2f034cff4c6423092cedc00c00c7159693"
Sep 08 14:02:37.330127 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:37.329944    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn" podUID="c969b297-3352-4bc2-8a5e-1a464927c074" containerName="main" containerID="cri-o://b9ffbd60d01e7bbf0fa408a46241eb2f034cff4c6423092cedc00c00c7159693" gracePeriod=60
Sep 08 14:02:37.499881 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:37.499830    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn" podUID="c969b297-3352-4bc2-8a5e-1a464927c074" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Sep 08 14:02:37.695570 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:37.695538    2558 generic.go:356] "Generic (PLEG): container finished" podID="c969b297-3352-4bc2-8a5e-1a464927c074" containerID="b9ffbd60d01e7bbf0fa408a46241eb2f034cff4c6423092cedc00c00c7159693" exitCode=0
Sep 08 14:02:37.695769 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:37.695628    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn" event={"ID":"c969b297-3352-4bc2-8a5e-1a464927c074","Type":"ContainerDied","Data":"b9ffbd60d01e7bbf0fa408a46241eb2f034cff4c6423092cedc00c00c7159693"}
Sep 08 14:02:38.070761 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:38.070727    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn"
Sep 08 14:02:38.131695 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:38.131652    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c969b297-3352-4bc2-8a5e-1a464927c074-model-cache\") pod \"c969b297-3352-4bc2-8a5e-1a464927c074\" (UID: \"c969b297-3352-4bc2-8a5e-1a464927c074\") "
Sep 08 14:02:38.131873 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:38.131707    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c969b297-3352-4bc2-8a5e-1a464927c074-tmp-dir\") pod \"c969b297-3352-4bc2-8a5e-1a464927c074\" (UID: \"c969b297-3352-4bc2-8a5e-1a464927c074\") "
Sep 08 14:02:38.131873 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:38.131764    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c969b297-3352-4bc2-8a5e-1a464927c074-dshm\") pod \"c969b297-3352-4bc2-8a5e-1a464927c074\" (UID: \"c969b297-3352-4bc2-8a5e-1a464927c074\") "
Sep 08 14:02:38.131873 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:38.131787    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c969b297-3352-4bc2-8a5e-1a464927c074-home\") pod \"c969b297-3352-4bc2-8a5e-1a464927c074\" (UID: \"c969b297-3352-4bc2-8a5e-1a464927c074\") "
Sep 08 14:02:38.131873 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:38.131819    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c969b297-3352-4bc2-8a5e-1a464927c074-tls-certs\") pod \"c969b297-3352-4bc2-8a5e-1a464927c074\" (UID: \"c969b297-3352-4bc2-8a5e-1a464927c074\") "
Sep 08 14:02:38.131873 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:38.131839    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lw62m\" (UniqueName: \"kubernetes.io/projected/c969b297-3352-4bc2-8a5e-1a464927c074-kube-api-access-lw62m\") pod \"c969b297-3352-4bc2-8a5e-1a464927c074\" (UID: \"c969b297-3352-4bc2-8a5e-1a464927c074\") "
Sep 08 14:02:38.132123 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:38.132027    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c969b297-3352-4bc2-8a5e-1a464927c074-model-cache" (OuterVolumeSpecName: "model-cache") pod "c969b297-3352-4bc2-8a5e-1a464927c074" (UID: "c969b297-3352-4bc2-8a5e-1a464927c074"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:02:38.132123 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:38.132069    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c969b297-3352-4bc2-8a5e-1a464927c074-home" (OuterVolumeSpecName: "home") pod "c969b297-3352-4bc2-8a5e-1a464927c074" (UID: "c969b297-3352-4bc2-8a5e-1a464927c074"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:02:38.132123 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:38.132076    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c969b297-3352-4bc2-8a5e-1a464927c074-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c969b297-3352-4bc2-8a5e-1a464927c074" (UID: "c969b297-3352-4bc2-8a5e-1a464927c074"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:02:38.132294 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:38.132230    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c969b297-3352-4bc2-8a5e-1a464927c074-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:02:38.132294 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:38.132251    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c969b297-3352-4bc2-8a5e-1a464927c074-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:02:38.132294 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:38.132268    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c969b297-3352-4bc2-8a5e-1a464927c074-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:02:38.134183 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:38.134153    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c969b297-3352-4bc2-8a5e-1a464927c074-dshm" (OuterVolumeSpecName: "dshm") pod "c969b297-3352-4bc2-8a5e-1a464927c074" (UID: "c969b297-3352-4bc2-8a5e-1a464927c074"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:02:38.134302 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:38.134210    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c969b297-3352-4bc2-8a5e-1a464927c074-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c969b297-3352-4bc2-8a5e-1a464927c074" (UID: "c969b297-3352-4bc2-8a5e-1a464927c074"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:02:38.134302 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:38.134220    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c969b297-3352-4bc2-8a5e-1a464927c074-kube-api-access-lw62m" (OuterVolumeSpecName: "kube-api-access-lw62m") pod "c969b297-3352-4bc2-8a5e-1a464927c074" (UID: "c969b297-3352-4bc2-8a5e-1a464927c074"). InnerVolumeSpecName "kube-api-access-lw62m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:02:38.233706 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:38.233666    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c969b297-3352-4bc2-8a5e-1a464927c074-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:02:38.233706 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:38.233700    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c969b297-3352-4bc2-8a5e-1a464927c074-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:02:38.233706 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:38.233711    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw62m\" (UniqueName: \"kubernetes.io/projected/c969b297-3352-4bc2-8a5e-1a464927c074-kube-api-access-lw62m\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:02:38.701980 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:38.701942    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn" event={"ID":"c969b297-3352-4bc2-8a5e-1a464927c074","Type":"ContainerDied","Data":"ee4a9dabaee99f03641f62686233abc48c41fea303130a1b3bcbcfa8b975df77"}
Sep 08 14:02:38.703776 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:38.702391    2558 scope.go:117] "RemoveContainer" containerID="b9ffbd60d01e7bbf0fa408a46241eb2f034cff4c6423092cedc00c00c7159693"
Sep 08 14:02:38.703776 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:38.702486    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn"
Sep 08 14:02:38.727686 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:38.727655    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn"]
Sep 08 14:02:38.730786 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:38.730765    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997tvkqn"]
Sep 08 14:02:39.057017 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:39.056936    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c969b297-3352-4bc2-8a5e-1a464927c074" path="/var/lib/kubelet/pods/c969b297-3352-4bc2-8a5e-1a464927c074/volumes"
Sep 08 14:02:50.209309 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.209266    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"]
Sep 08 14:02:50.209827 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.209667    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c969b297-3352-4bc2-8a5e-1a464927c074" containerName="main"
Sep 08 14:02:50.209827 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.209680    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c969b297-3352-4bc2-8a5e-1a464927c074" containerName="main"
Sep 08 14:02:50.209827 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.209754    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c969b297-3352-4bc2-8a5e-1a464927c074" containerName="main"
Sep 08 14:02:50.212578 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.212559    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"
Sep 08 14:02:50.215973 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.215934    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 08 14:02:50.216131 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.215936    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Sep 08 14:02:50.216131 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.216038    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 14:02:50.216131 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.216005    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-qzxkd\""
Sep 08 14:02:50.216131 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.216077    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 08 14:02:50.216315 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.216150    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 08 14:02:50.229528 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.229500    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"]
Sep 08 14:02:50.342175 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.342126    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c2465e1-9292-4d96-99cb-183044999ee8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt\" (UID: \"6c2465e1-9292-4d96-99cb-183044999ee8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"
Sep 08 14:02:50.342175 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.342178    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqlb8\" (UniqueName: \"kubernetes.io/projected/6c2465e1-9292-4d96-99cb-183044999ee8-kube-api-access-sqlb8\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt\" (UID: \"6c2465e1-9292-4d96-99cb-183044999ee8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"
Sep 08 14:02:50.342421 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.342207    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c2465e1-9292-4d96-99cb-183044999ee8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt\" (UID: \"6c2465e1-9292-4d96-99cb-183044999ee8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"
Sep 08 14:02:50.342421 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.342240    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c2465e1-9292-4d96-99cb-183044999ee8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt\" (UID: \"6c2465e1-9292-4d96-99cb-183044999ee8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"
Sep 08 14:02:50.342421 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.342268    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c2465e1-9292-4d96-99cb-183044999ee8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt\" (UID: \"6c2465e1-9292-4d96-99cb-183044999ee8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"
Sep 08 14:02:50.342421 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.342311    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c2465e1-9292-4d96-99cb-183044999ee8-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt\" (UID: \"6c2465e1-9292-4d96-99cb-183044999ee8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"
Sep 08 14:02:50.342421 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.342356    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c2465e1-9292-4d96-99cb-183044999ee8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt\" (UID: \"6c2465e1-9292-4d96-99cb-183044999ee8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"
Sep 08 14:02:50.342421 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.342392    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c2465e1-9292-4d96-99cb-183044999ee8-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt\" (UID: \"6c2465e1-9292-4d96-99cb-183044999ee8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"
Sep 08 14:02:50.443104 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.443068    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c2465e1-9292-4d96-99cb-183044999ee8-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt\" (UID: \"6c2465e1-9292-4d96-99cb-183044999ee8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"
Sep 08 14:02:50.443285 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.443234    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c2465e1-9292-4d96-99cb-183044999ee8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt\" (UID: \"6c2465e1-9292-4d96-99cb-183044999ee8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"
Sep 08 14:02:50.443335 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.443293    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqlb8\" (UniqueName: \"kubernetes.io/projected/6c2465e1-9292-4d96-99cb-183044999ee8-kube-api-access-sqlb8\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt\" (UID: \"6c2465e1-9292-4d96-99cb-183044999ee8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"
Sep 08 14:02:50.443335 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.443325    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c2465e1-9292-4d96-99cb-183044999ee8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt\" (UID: \"6c2465e1-9292-4d96-99cb-183044999ee8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"
Sep 08 14:02:50.443418 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.443353    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c2465e1-9292-4d96-99cb-183044999ee8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt\" (UID: \"6c2465e1-9292-4d96-99cb-183044999ee8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"
Sep 08 14:02:50.443506 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.443480    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c2465e1-9292-4d96-99cb-183044999ee8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt\" (UID: \"6c2465e1-9292-4d96-99cb-183044999ee8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"
Sep 08 14:02:50.443570 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.443526    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c2465e1-9292-4d96-99cb-183044999ee8-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt\" (UID: \"6c2465e1-9292-4d96-99cb-183044999ee8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"
Sep 08 14:02:50.443649 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.443579    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c2465e1-9292-4d96-99cb-183044999ee8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt\" (UID: \"6c2465e1-9292-4d96-99cb-183044999ee8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"
Sep 08 14:02:50.443649 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.443628    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c2465e1-9292-4d96-99cb-183044999ee8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt\" (UID: \"6c2465e1-9292-4d96-99cb-183044999ee8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"
Sep 08 14:02:50.443776 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.443678    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c2465e1-9292-4d96-99cb-183044999ee8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt\" (UID: \"6c2465e1-9292-4d96-99cb-183044999ee8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"
Sep 08 14:02:50.443893 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.443863    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c2465e1-9292-4d96-99cb-183044999ee8-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt\" (UID: \"6c2465e1-9292-4d96-99cb-183044999ee8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"
Sep 08 14:02:50.443982 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.443902    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c2465e1-9292-4d96-99cb-183044999ee8-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt\" (UID: \"6c2465e1-9292-4d96-99cb-183044999ee8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"
Sep 08 14:02:50.443982 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.443907    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c2465e1-9292-4d96-99cb-183044999ee8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt\" (UID: \"6c2465e1-9292-4d96-99cb-183044999ee8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"
Sep 08 14:02:50.445752 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.445729    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c2465e1-9292-4d96-99cb-183044999ee8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt\" (UID: \"6c2465e1-9292-4d96-99cb-183044999ee8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"
Sep 08 14:02:50.446002 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.445984    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c2465e1-9292-4d96-99cb-183044999ee8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt\" (UID: \"6c2465e1-9292-4d96-99cb-183044999ee8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"
Sep 08 14:02:50.453569 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.453546    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqlb8\" (UniqueName: \"kubernetes.io/projected/6c2465e1-9292-4d96-99cb-183044999ee8-kube-api-access-sqlb8\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt\" (UID: \"6c2465e1-9292-4d96-99cb-183044999ee8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"
Sep 08 14:02:50.522712 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.522612    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"
Sep 08 14:02:50.805408 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:50.805324    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"]
Sep 08 14:02:50.808386 ip-10-0-128-147 kubenswrapper[2558]: W0908 14:02:50.808352    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c2465e1_9292_4d96_99cb_183044999ee8.slice/crio-8f7ba3c1350ebed99943a827d20e677ff0ab5dd3277545ecd626a2da4ad1c33b WatchSource:0}: Error finding container 8f7ba3c1350ebed99943a827d20e677ff0ab5dd3277545ecd626a2da4ad1c33b: Status 404 returned error can't find the container with id 8f7ba3c1350ebed99943a827d20e677ff0ab5dd3277545ecd626a2da4ad1c33b
Sep 08 14:02:51.767779 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:51.767736    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt" event={"ID":"6c2465e1-9292-4d96-99cb-183044999ee8","Type":"ContainerStarted","Data":"b7826e8b0a52622843802d6484aef6d02dcff11d342cf94bdb4680f136070073"}
Sep 08 14:02:51.768378 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:51.768343    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt" event={"ID":"6c2465e1-9292-4d96-99cb-183044999ee8","Type":"ContainerStarted","Data":"8f7ba3c1350ebed99943a827d20e677ff0ab5dd3277545ecd626a2da4ad1c33b"}
Sep 08 14:02:53.779765 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:53.779725    2558 generic.go:356] "Generic (PLEG): container finished" podID="6c2465e1-9292-4d96-99cb-183044999ee8" containerID="b7826e8b0a52622843802d6484aef6d02dcff11d342cf94bdb4680f136070073" exitCode=0
Sep 08 14:02:53.779765 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:53.779767    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt" event={"ID":"6c2465e1-9292-4d96-99cb-183044999ee8","Type":"ContainerDied","Data":"b7826e8b0a52622843802d6484aef6d02dcff11d342cf94bdb4680f136070073"}
Sep 08 14:02:54.786876 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:54.786840    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt" event={"ID":"6c2465e1-9292-4d96-99cb-183044999ee8","Type":"ContainerStarted","Data":"7896ded848bf515499f0046832fac21c417279879b5bf3141fd87ea76c4f4d38"}
Sep 08 14:02:54.811094 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:54.811007    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt" podStartSLOduration=2.564774045 podStartE2EDuration="4.810985351s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.246211306s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:02:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:02:54.810195503 +0000 UTC m=+2838.261956473" watchObservedRunningTime="2026-09-08 14:02:54.810985351 +0000 UTC m=+2838.262746281"
Sep 08 14:02:55.523393 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:02:55.523357    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"
Sep 08 14:03:00.523078 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:03:00.523043    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"
Sep 08 14:03:00.524622 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:03:00.524575    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt" podUID="6c2465e1-9292-4d96-99cb-183044999ee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 08 14:03:10.523738 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:03:10.523677    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt" podUID="6c2465e1-9292-4d96-99cb-183044999ee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 08 14:03:20.523341 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:03:20.523293    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt" podUID="6c2465e1-9292-4d96-99cb-183044999ee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 08 14:03:30.522947 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:03:30.522902    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt" podUID="6c2465e1-9292-4d96-99cb-183044999ee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 08 14:03:40.524009 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:03:40.523959    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt" podUID="6c2465e1-9292-4d96-99cb-183044999ee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 08 14:03:50.523516 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:03:50.523474    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt" podUID="6c2465e1-9292-4d96-99cb-183044999ee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 08 14:04:00.523008 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:04:00.522958    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt" podUID="6c2465e1-9292-4d96-99cb-183044999ee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 08 14:04:10.524114 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:04:10.524065    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt" podUID="6c2465e1-9292-4d96-99cb-183044999ee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 08 14:04:20.523742 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:04:20.523679    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt" podUID="6c2465e1-9292-4d96-99cb-183044999ee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 08 14:04:30.523089 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:04:30.523043    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt" podUID="6c2465e1-9292-4d96-99cb-183044999ee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 08 14:04:40.523683 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:04:40.523631    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt" podUID="6c2465e1-9292-4d96-99cb-183044999ee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 08 14:04:50.532890 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:04:50.532855    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"
Sep 08 14:04:50.540888 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:04:50.540864    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"
Sep 08 14:04:55.622627 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:04:55.622572    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"]
Sep 08 14:05:10.646001 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:10.645887    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt" podUID="6c2465e1-9292-4d96-99cb-183044999ee8" containerName="main" containerID="cri-o://7896ded848bf515499f0046832fac21c417279879b5bf3141fd87ea76c4f4d38" gracePeriod=45
Sep 08 14:05:11.394836 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.394809    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"
Sep 08 14:05:11.496303 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.496216    2558 generic.go:356] "Generic (PLEG): container finished" podID="6c2465e1-9292-4d96-99cb-183044999ee8" containerID="7896ded848bf515499f0046832fac21c417279879b5bf3141fd87ea76c4f4d38" exitCode=0
Sep 08 14:05:11.496303 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.496263    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"
Sep 08 14:05:11.496542 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.496261    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt" event={"ID":"6c2465e1-9292-4d96-99cb-183044999ee8","Type":"ContainerDied","Data":"7896ded848bf515499f0046832fac21c417279879b5bf3141fd87ea76c4f4d38"}
Sep 08 14:05:11.496542 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.496381    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt" event={"ID":"6c2465e1-9292-4d96-99cb-183044999ee8","Type":"ContainerDied","Data":"8f7ba3c1350ebed99943a827d20e677ff0ab5dd3277545ecd626a2da4ad1c33b"}
Sep 08 14:05:11.496542 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.496404    2558 scope.go:117] "RemoveContainer" containerID="7896ded848bf515499f0046832fac21c417279879b5bf3141fd87ea76c4f4d38"
Sep 08 14:05:11.504789 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.504772    2558 scope.go:117] "RemoveContainer" containerID="b7826e8b0a52622843802d6484aef6d02dcff11d342cf94bdb4680f136070073"
Sep 08 14:05:11.512501 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.512483    2558 scope.go:117] "RemoveContainer" containerID="7896ded848bf515499f0046832fac21c417279879b5bf3141fd87ea76c4f4d38"
Sep 08 14:05:11.512822 ip-10-0-128-147 kubenswrapper[2558]: E0908 14:05:11.512800    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7896ded848bf515499f0046832fac21c417279879b5bf3141fd87ea76c4f4d38\": container with ID starting with 7896ded848bf515499f0046832fac21c417279879b5bf3141fd87ea76c4f4d38 not found: ID does not exist" containerID="7896ded848bf515499f0046832fac21c417279879b5bf3141fd87ea76c4f4d38"
Sep 08 14:05:11.512872 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.512836    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7896ded848bf515499f0046832fac21c417279879b5bf3141fd87ea76c4f4d38"} err="failed to get container status \"7896ded848bf515499f0046832fac21c417279879b5bf3141fd87ea76c4f4d38\": rpc error: code = NotFound desc = could not find container \"7896ded848bf515499f0046832fac21c417279879b5bf3141fd87ea76c4f4d38\": container with ID starting with 7896ded848bf515499f0046832fac21c417279879b5bf3141fd87ea76c4f4d38 not found: ID does not exist"
Sep 08 14:05:11.512872 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.512856    2558 scope.go:117] "RemoveContainer" containerID="b7826e8b0a52622843802d6484aef6d02dcff11d342cf94bdb4680f136070073"
Sep 08 14:05:11.513133 ip-10-0-128-147 kubenswrapper[2558]: E0908 14:05:11.513113    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7826e8b0a52622843802d6484aef6d02dcff11d342cf94bdb4680f136070073\": container with ID starting with b7826e8b0a52622843802d6484aef6d02dcff11d342cf94bdb4680f136070073 not found: ID does not exist" containerID="b7826e8b0a52622843802d6484aef6d02dcff11d342cf94bdb4680f136070073"
Sep 08 14:05:11.513185 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.513140    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7826e8b0a52622843802d6484aef6d02dcff11d342cf94bdb4680f136070073"} err="failed to get container status \"b7826e8b0a52622843802d6484aef6d02dcff11d342cf94bdb4680f136070073\": rpc error: code = NotFound desc = could not find container \"b7826e8b0a52622843802d6484aef6d02dcff11d342cf94bdb4680f136070073\": container with ID starting with b7826e8b0a52622843802d6484aef6d02dcff11d342cf94bdb4680f136070073 not found: ID does not exist"
Sep 08 14:05:11.513257 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.513244    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c2465e1-9292-4d96-99cb-183044999ee8-tmp-dir\") pod \"6c2465e1-9292-4d96-99cb-183044999ee8\" (UID: \"6c2465e1-9292-4d96-99cb-183044999ee8\") "
Sep 08 14:05:11.513297 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.513276    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c2465e1-9292-4d96-99cb-183044999ee8-tls-certs\") pod \"6c2465e1-9292-4d96-99cb-183044999ee8\" (UID: \"6c2465e1-9292-4d96-99cb-183044999ee8\") "
Sep 08 14:05:11.513331 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.513314    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c2465e1-9292-4d96-99cb-183044999ee8-model-cache\") pod \"6c2465e1-9292-4d96-99cb-183044999ee8\" (UID: \"6c2465e1-9292-4d96-99cb-183044999ee8\") "
Sep 08 14:05:11.513369 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.513341    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c2465e1-9292-4d96-99cb-183044999ee8-cabundle-cert\") pod \"6c2465e1-9292-4d96-99cb-183044999ee8\" (UID: \"6c2465e1-9292-4d96-99cb-183044999ee8\") "
Sep 08 14:05:11.513403 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.513373    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c2465e1-9292-4d96-99cb-183044999ee8-dshm\") pod \"6c2465e1-9292-4d96-99cb-183044999ee8\" (UID: \"6c2465e1-9292-4d96-99cb-183044999ee8\") "
Sep 08 14:05:11.513447 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.513402    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqlb8\" (UniqueName: \"kubernetes.io/projected/6c2465e1-9292-4d96-99cb-183044999ee8-kube-api-access-sqlb8\") pod \"6c2465e1-9292-4d96-99cb-183044999ee8\" (UID: \"6c2465e1-9292-4d96-99cb-183044999ee8\") "
Sep 08 14:05:11.513498 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.513465    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c2465e1-9292-4d96-99cb-183044999ee8-kserve-provision-location\") pod \"6c2465e1-9292-4d96-99cb-183044999ee8\" (UID: \"6c2465e1-9292-4d96-99cb-183044999ee8\") "
Sep 08 14:05:11.513554 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.513510    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c2465e1-9292-4d96-99cb-183044999ee8-home\") pod \"6c2465e1-9292-4d96-99cb-183044999ee8\" (UID: \"6c2465e1-9292-4d96-99cb-183044999ee8\") "
Sep 08 14:05:11.513656 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.513629    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2465e1-9292-4d96-99cb-183044999ee8-model-cache" (OuterVolumeSpecName: "model-cache") pod "6c2465e1-9292-4d96-99cb-183044999ee8" (UID: "6c2465e1-9292-4d96-99cb-183044999ee8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:05:11.513739 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.513699    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c2465e1-9292-4d96-99cb-183044999ee8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6c2465e1-9292-4d96-99cb-183044999ee8" (UID: "6c2465e1-9292-4d96-99cb-183044999ee8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:05:11.513879 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.513851    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c2465e1-9292-4d96-99cb-183044999ee8-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:05:11.513974 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.513889    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c2465e1-9292-4d96-99cb-183044999ee8-cabundle-cert\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:05:11.514352 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.514326    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2465e1-9292-4d96-99cb-183044999ee8-home" (OuterVolumeSpecName: "home") pod "6c2465e1-9292-4d96-99cb-183044999ee8" (UID: "6c2465e1-9292-4d96-99cb-183044999ee8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:05:11.516220 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.516192    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2465e1-9292-4d96-99cb-183044999ee8-dshm" (OuterVolumeSpecName: "dshm") pod "6c2465e1-9292-4d96-99cb-183044999ee8" (UID: "6c2465e1-9292-4d96-99cb-183044999ee8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:05:11.516336 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.516233    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c2465e1-9292-4d96-99cb-183044999ee8-kube-api-access-sqlb8" (OuterVolumeSpecName: "kube-api-access-sqlb8") pod "6c2465e1-9292-4d96-99cb-183044999ee8" (UID: "6c2465e1-9292-4d96-99cb-183044999ee8"). InnerVolumeSpecName "kube-api-access-sqlb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:05:11.516336 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.516305    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c2465e1-9292-4d96-99cb-183044999ee8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6c2465e1-9292-4d96-99cb-183044999ee8" (UID: "6c2465e1-9292-4d96-99cb-183044999ee8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:05:11.526160 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.526130    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2465e1-9292-4d96-99cb-183044999ee8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6c2465e1-9292-4d96-99cb-183044999ee8" (UID: "6c2465e1-9292-4d96-99cb-183044999ee8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:05:11.536167 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.536140    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2465e1-9292-4d96-99cb-183044999ee8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6c2465e1-9292-4d96-99cb-183044999ee8" (UID: "6c2465e1-9292-4d96-99cb-183044999ee8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:05:11.615132 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.615096    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c2465e1-9292-4d96-99cb-183044999ee8-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:05:11.615132 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.615130    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqlb8\" (UniqueName: \"kubernetes.io/projected/6c2465e1-9292-4d96-99cb-183044999ee8-kube-api-access-sqlb8\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:05:11.615348 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.615144    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c2465e1-9292-4d96-99cb-183044999ee8-kserve-provision-location\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:05:11.615348 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.615158    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c2465e1-9292-4d96-99cb-183044999ee8-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:05:11.615348 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.615173    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c2465e1-9292-4d96-99cb-183044999ee8-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:05:11.615348 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.615183    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c2465e1-9292-4d96-99cb-183044999ee8-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:05:11.818529 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.818496    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"]
Sep 08 14:05:11.821785 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:11.821757    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b654d78cdf2dqt"]
Sep 08 14:05:13.053120 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:05:13.053082    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c2465e1-9292-4d96-99cb-183044999ee8" path="/var/lib/kubelet/pods/6c2465e1-9292-4d96-99cb-183044999ee8/volumes"
Sep 08 14:06:03.911241 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:03.911207    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"]
Sep 08 14:06:03.911962 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:03.911924    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c2465e1-9292-4d96-99cb-183044999ee8" containerName="main"
Sep 08 14:06:03.911962 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:03.911953    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c2465e1-9292-4d96-99cb-183044999ee8" containerName="main"
Sep 08 14:06:03.912172 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:03.911998    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c2465e1-9292-4d96-99cb-183044999ee8" containerName="storage-initializer"
Sep 08 14:06:03.912172 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:03.912008    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c2465e1-9292-4d96-99cb-183044999ee8" containerName="storage-initializer"
Sep 08 14:06:03.912426 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:03.912400    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c2465e1-9292-4d96-99cb-183044999ee8" containerName="main"
Sep 08 14:06:03.919386 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:03.919360    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"
Sep 08 14:06:03.922152 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:03.922110    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 08 14:06:03.922329 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:03.922204    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 08 14:06:03.922458 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:03.922254    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 08 14:06:03.922458 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:03.922262    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-kjh6z\""
Sep 08 14:06:03.927906 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:03.927885    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"]
Sep 08 14:06:04.064301 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:04.064264    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7\" (UID: \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"
Sep 08 14:06:04.064478 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:04.064307    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7\" (UID: \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"
Sep 08 14:06:04.064478 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:04.064379    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7\" (UID: \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"
Sep 08 14:06:04.064478 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:04.064419    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7\" (UID: \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"
Sep 08 14:06:04.064478 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:04.064447    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7\" (UID: \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"
Sep 08 14:06:04.064478 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:04.064464    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7\" (UID: \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"
Sep 08 14:06:04.064690 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:04.064515    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rd2z\" (UniqueName: \"kubernetes.io/projected/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-kube-api-access-8rd2z\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7\" (UID: \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"
Sep 08 14:06:04.165646 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:04.165530    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7\" (UID: \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"
Sep 08 14:06:04.165646 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:04.165580    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7\" (UID: \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"
Sep 08 14:06:04.165893 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:04.165648    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7\" (UID: \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"
Sep 08 14:06:04.165893 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:04.165672    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7\" (UID: \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"
Sep 08 14:06:04.165893 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:04.165704    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7\" (UID: \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"
Sep 08 14:06:04.165893 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:04.165728    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7\" (UID: \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"
Sep 08 14:06:04.165893 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:04.165844    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rd2z\" (UniqueName: \"kubernetes.io/projected/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-kube-api-access-8rd2z\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7\" (UID: \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"
Sep 08 14:06:04.166117 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:04.166021    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7\" (UID: \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"
Sep 08 14:06:04.166117 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:04.166061    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7\" (UID: \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"
Sep 08 14:06:04.166206 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:04.166152    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7\" (UID: \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"
Sep 08 14:06:04.166280 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:04.166265    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7\" (UID: \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"
Sep 08 14:06:04.168071 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:04.168050    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7\" (UID: \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"
Sep 08 14:06:04.168252 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:04.168236    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7\" (UID: \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"
Sep 08 14:06:04.178078 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:04.178056    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rd2z\" (UniqueName: \"kubernetes.io/projected/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-kube-api-access-8rd2z\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7\" (UID: \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"
Sep 08 14:06:04.230764 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:04.230731    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"
Sep 08 14:06:04.510179 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:04.510151    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"]
Sep 08 14:06:04.555812 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:04.555788    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 14:06:04.748638 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:04.748585    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7" event={"ID":"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b","Type":"ContainerStarted","Data":"dd82d8dc0e8e5c9cde061697152279b53f75950a2b08d6e2b7e526822f8af924"}
Sep 08 14:06:13.803496 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:13.803456    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7" event={"ID":"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b","Type":"ContainerStarted","Data":"23f3a16599c7b6e9946a66e21103bd7c2ddd094279c49248011a7c3a5ef6f330"}
Sep 08 14:06:14.810033 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:14.809990    2558 generic.go:356] "Generic (PLEG): container finished" podID="f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b" containerID="23f3a16599c7b6e9946a66e21103bd7c2ddd094279c49248011a7c3a5ef6f330" exitCode=0
Sep 08 14:06:14.810435 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:14.810074    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7" event={"ID":"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b","Type":"ContainerDied","Data":"23f3a16599c7b6e9946a66e21103bd7c2ddd094279c49248011a7c3a5ef6f330"}
Sep 08 14:06:15.823749 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:15.823655    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7" event={"ID":"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b","Type":"ContainerStarted","Data":"35c07303748d7ef5d2223e8b3714a4fc95214e1ff4d5bd9f90166132be3dae44"}
Sep 08 14:06:15.824141 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:15.823776    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7" event={"ID":"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b","Type":"ContainerStarted","Data":"91b56b04851fd3e11628369d2d155e3ae79293b239ee9f6f44ca9e0bf712959c"}
Sep 08 14:06:15.848443 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:15.848372    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7" podStartSLOduration=3.305136065 podStartE2EDuration="12.848349086s" totalImagesPullingTime="8.456620842s" totalInitContainerRuntime="1.086592179s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:06:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:06:15.846017729 +0000 UTC m=+3039.297778660" watchObservedRunningTime="2026-09-08 14:06:15.848349086 +0000 UTC m=+3039.300110018"
Sep 08 14:06:16.231115 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:16.231079    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"
Sep 08 14:06:24.231369 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:24.231326    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"
Sep 08 14:06:24.233006 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:24.232970    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7" podUID="f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 08 14:06:34.231586 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:34.231493    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7" podUID="f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 08 14:06:44.231235 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:44.231187    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7" podUID="f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 08 14:06:54.231822 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:06:54.231766    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7" podUID="f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 08 14:07:04.231428 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:07:04.231381    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7" podUID="f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 08 14:07:14.231996 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:07:14.231946    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7" podUID="f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 08 14:07:24.232178 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:07:24.232127    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7" podUID="f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 08 14:07:34.231608 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:07:34.231540    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7" podUID="f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 08 14:07:44.231795 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:07:44.231736    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7" podUID="f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 08 14:07:54.231705 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:07:54.231657    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7" podUID="f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 08 14:08:04.231780 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:04.231675    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7" podUID="f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 08 14:08:14.241662 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:14.241619    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"
Sep 08 14:08:14.254753 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:14.254714    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"
Sep 08 14:08:19.593884 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:19.593852    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"]
Sep 08 14:08:19.594314 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:19.594187    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7" podUID="f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b" containerName="modelcar" containerID="cri-o://35c07303748d7ef5d2223e8b3714a4fc95214e1ff4d5bd9f90166132be3dae44" gracePeriod=60
Sep 08 14:08:20.461564 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:20.461532    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7_f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b/modelcar/0.log"
Sep 08 14:08:20.462256 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:20.462229    2558 generic.go:356] "Generic (PLEG): container finished" podID="f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b" containerID="35c07303748d7ef5d2223e8b3714a4fc95214e1ff4d5bd9f90166132be3dae44" exitCode=143
Sep 08 14:08:20.462322 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:20.462280    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7" event={"ID":"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b","Type":"ContainerDied","Data":"35c07303748d7ef5d2223e8b3714a4fc95214e1ff4d5bd9f90166132be3dae44"}
Sep 08 14:08:34.622251 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:34.622182    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7" podUID="f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b" containerName="main" containerID="cri-o://91b56b04851fd3e11628369d2d155e3ae79293b239ee9f6f44ca9e0bf712959c" gracePeriod=45
Sep 08 14:08:35.243102 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:35.243054    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7" podUID="f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 08 14:08:35.531011 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:35.530926    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7_f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b/modelcar/0.log"
Sep 08 14:08:35.531683 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:35.531657    2558 generic.go:356] "Generic (PLEG): container finished" podID="f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b" containerID="91b56b04851fd3e11628369d2d155e3ae79293b239ee9f6f44ca9e0bf712959c" exitCode=0
Sep 08 14:08:35.531783 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:35.531725    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7" event={"ID":"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b","Type":"ContainerDied","Data":"91b56b04851fd3e11628369d2d155e3ae79293b239ee9f6f44ca9e0bf712959c"}
Sep 08 14:08:35.874367 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:35.874341    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7_f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b/modelcar/0.log"
Sep 08 14:08:35.877264 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:35.877245    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"
Sep 08 14:08:35.938887 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:35.938854    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-tmp-dir\") pod \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\" (UID: \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\") "
Sep 08 14:08:35.939047 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:35.938952    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-kserve-provision-location\") pod \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\" (UID: \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\") "
Sep 08 14:08:35.939047 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:35.938990    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-dshm\") pod \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\" (UID: \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\") "
Sep 08 14:08:35.939047 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:35.939017    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-model-cache\") pod \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\" (UID: \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\") "
Sep 08 14:08:35.939047 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:35.939043    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rd2z\" (UniqueName: \"kubernetes.io/projected/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-kube-api-access-8rd2z\") pod \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\" (UID: \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\") "
Sep 08 14:08:35.939255 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:35.939077    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-home\") pod \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\" (UID: \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\") "
Sep 08 14:08:35.939255 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:35.939100    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-tls-certs\") pod \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\" (UID: \"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b\") "
Sep 08 14:08:35.939380 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:35.939265    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-model-cache" (OuterVolumeSpecName: "model-cache") pod "f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b" (UID: "f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:08:35.939380 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:35.939308    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b" (UID: "f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:08:35.939497 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:35.939401    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-kserve-provision-location\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:08:35.939497 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:35.939422    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:08:35.939698 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:35.939673    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-home" (OuterVolumeSpecName: "home") pod "f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b" (UID: "f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:08:35.941304 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:35.941277    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-dshm" (OuterVolumeSpecName: "dshm") pod "f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b" (UID: "f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:08:35.941412 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:35.941336    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b" (UID: "f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:08:35.941667 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:35.941642    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-kube-api-access-8rd2z" (OuterVolumeSpecName: "kube-api-access-8rd2z") pod "f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b" (UID: "f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b"). InnerVolumeSpecName "kube-api-access-8rd2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:08:35.950831 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:35.950802    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b" (UID: "f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:08:36.040296 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:36.040262    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:08:36.040296 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:36.040293    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rd2z\" (UniqueName: \"kubernetes.io/projected/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-kube-api-access-8rd2z\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:08:36.040296 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:36.040304    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:08:36.040529 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:36.040313    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:08:36.040529 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:36.040344    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:08:36.538010 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:36.537981    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7_f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b/modelcar/0.log"
Sep 08 14:08:36.538760 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:36.538727    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7" event={"ID":"f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b","Type":"ContainerDied","Data":"dd82d8dc0e8e5c9cde061697152279b53f75950a2b08d6e2b7e526822f8af924"}
Sep 08 14:08:36.538879 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:36.538759    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"
Sep 08 14:08:36.538879 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:36.538802    2558 scope.go:117] "RemoveContainer" containerID="35c07303748d7ef5d2223e8b3714a4fc95214e1ff4d5bd9f90166132be3dae44"
Sep 08 14:08:36.548563 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:36.548543    2558 scope.go:117] "RemoveContainer" containerID="91b56b04851fd3e11628369d2d155e3ae79293b239ee9f6f44ca9e0bf712959c"
Sep 08 14:08:36.556746 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:36.556727    2558 scope.go:117] "RemoveContainer" containerID="23f3a16599c7b6e9946a66e21103bd7c2ddd094279c49248011a7c3a5ef6f330"
Sep 08 14:08:36.562285 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:36.562263    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"]
Sep 08 14:08:36.571821 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:36.571797    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-6f98cf8b8f9k6k7"]
Sep 08 14:08:37.055371 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:08:37.055332    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b" path="/var/lib/kubelet/pods/f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b/volumes"
Sep 08 14:09:37.724778 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.724741    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"]
Sep 08 14:09:37.725217 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.725147    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b" containerName="main"
Sep 08 14:09:37.725217 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.725178    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b" containerName="main"
Sep 08 14:09:37.725217 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.725190    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b" containerName="modelcar"
Sep 08 14:09:37.725217 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.725195    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b" containerName="modelcar"
Sep 08 14:09:37.725217 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.725203    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b" containerName="modelcar-init"
Sep 08 14:09:37.725217 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.725209    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b" containerName="modelcar-init"
Sep 08 14:09:37.725412 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.725265    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b" containerName="main"
Sep 08 14:09:37.725412 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.725277    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7fcb326-2e5e-42f6-8ac2-7f0ae99dcd5b" containerName="modelcar"
Sep 08 14:09:37.728371 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.728353    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"
Sep 08 14:09:37.730770 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.730743    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 08 14:09:37.730916 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.730845    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-dl8fm\""
Sep 08 14:09:37.730916 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.730845    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 08 14:09:37.731322 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.731306    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 08 14:09:37.731487 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.731462    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Sep 08 14:09:37.731581 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.731525    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 14:09:37.745017 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.744986    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"]
Sep 08 14:09:37.775911 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.775878    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32af396a-976c-4f92-ab41-e0038cb69550-dshm\") pod \"custom-route-timeout-test-kserve-84bf8f4d7f-htznc\" (UID: \"32af396a-976c-4f92-ab41-e0038cb69550\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"
Sep 08 14:09:37.775911 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.775915    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32af396a-976c-4f92-ab41-e0038cb69550-model-cache\") pod \"custom-route-timeout-test-kserve-84bf8f4d7f-htznc\" (UID: \"32af396a-976c-4f92-ab41-e0038cb69550\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"
Sep 08 14:09:37.776112 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.775937    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32af396a-976c-4f92-ab41-e0038cb69550-tmp-dir\") pod \"custom-route-timeout-test-kserve-84bf8f4d7f-htznc\" (UID: \"32af396a-976c-4f92-ab41-e0038cb69550\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"
Sep 08 14:09:37.776112 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.776037    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32af396a-976c-4f92-ab41-e0038cb69550-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-84bf8f4d7f-htznc\" (UID: \"32af396a-976c-4f92-ab41-e0038cb69550\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"
Sep 08 14:09:37.776112 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.776072    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32af396a-976c-4f92-ab41-e0038cb69550-cabundle-cert\") pod \"custom-route-timeout-test-kserve-84bf8f4d7f-htznc\" (UID: \"32af396a-976c-4f92-ab41-e0038cb69550\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"
Sep 08 14:09:37.776211 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.776154    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32af396a-976c-4f92-ab41-e0038cb69550-tls-certs\") pod \"custom-route-timeout-test-kserve-84bf8f4d7f-htznc\" (UID: \"32af396a-976c-4f92-ab41-e0038cb69550\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"
Sep 08 14:09:37.776211 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.776189    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32af396a-976c-4f92-ab41-e0038cb69550-home\") pod \"custom-route-timeout-test-kserve-84bf8f4d7f-htznc\" (UID: \"32af396a-976c-4f92-ab41-e0038cb69550\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"
Sep 08 14:09:37.776288 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.776220    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtmgm\" (UniqueName: \"kubernetes.io/projected/32af396a-976c-4f92-ab41-e0038cb69550-kube-api-access-rtmgm\") pod \"custom-route-timeout-test-kserve-84bf8f4d7f-htznc\" (UID: \"32af396a-976c-4f92-ab41-e0038cb69550\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"
Sep 08 14:09:37.877662 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.877629    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32af396a-976c-4f92-ab41-e0038cb69550-home\") pod \"custom-route-timeout-test-kserve-84bf8f4d7f-htznc\" (UID: \"32af396a-976c-4f92-ab41-e0038cb69550\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"
Sep 08 14:09:37.877874 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.877671    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtmgm\" (UniqueName: \"kubernetes.io/projected/32af396a-976c-4f92-ab41-e0038cb69550-kube-api-access-rtmgm\") pod \"custom-route-timeout-test-kserve-84bf8f4d7f-htznc\" (UID: \"32af396a-976c-4f92-ab41-e0038cb69550\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"
Sep 08 14:09:37.877874 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.877792    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32af396a-976c-4f92-ab41-e0038cb69550-dshm\") pod \"custom-route-timeout-test-kserve-84bf8f4d7f-htznc\" (UID: \"32af396a-976c-4f92-ab41-e0038cb69550\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"
Sep 08 14:09:37.877874 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.877843    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32af396a-976c-4f92-ab41-e0038cb69550-model-cache\") pod \"custom-route-timeout-test-kserve-84bf8f4d7f-htznc\" (UID: \"32af396a-976c-4f92-ab41-e0038cb69550\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"
Sep 08 14:09:37.877874 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.877867    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32af396a-976c-4f92-ab41-e0038cb69550-tmp-dir\") pod \"custom-route-timeout-test-kserve-84bf8f4d7f-htznc\" (UID: \"32af396a-976c-4f92-ab41-e0038cb69550\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"
Sep 08 14:09:37.878118 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.877927    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32af396a-976c-4f92-ab41-e0038cb69550-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-84bf8f4d7f-htznc\" (UID: \"32af396a-976c-4f92-ab41-e0038cb69550\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"
Sep 08 14:09:37.878118 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.877966    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32af396a-976c-4f92-ab41-e0038cb69550-cabundle-cert\") pod \"custom-route-timeout-test-kserve-84bf8f4d7f-htznc\" (UID: \"32af396a-976c-4f92-ab41-e0038cb69550\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"
Sep 08 14:09:37.878118 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.878023    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32af396a-976c-4f92-ab41-e0038cb69550-tls-certs\") pod \"custom-route-timeout-test-kserve-84bf8f4d7f-htznc\" (UID: \"32af396a-976c-4f92-ab41-e0038cb69550\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"
Sep 08 14:09:37.878118 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.878101    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32af396a-976c-4f92-ab41-e0038cb69550-home\") pod \"custom-route-timeout-test-kserve-84bf8f4d7f-htznc\" (UID: \"32af396a-976c-4f92-ab41-e0038cb69550\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"
Sep 08 14:09:37.878330 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.878284    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32af396a-976c-4f92-ab41-e0038cb69550-tmp-dir\") pod \"custom-route-timeout-test-kserve-84bf8f4d7f-htznc\" (UID: \"32af396a-976c-4f92-ab41-e0038cb69550\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"
Sep 08 14:09:37.878330 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.878311    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32af396a-976c-4f92-ab41-e0038cb69550-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-84bf8f4d7f-htznc\" (UID: \"32af396a-976c-4f92-ab41-e0038cb69550\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"
Sep 08 14:09:37.878719 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.878693    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32af396a-976c-4f92-ab41-e0038cb69550-cabundle-cert\") pod \"custom-route-timeout-test-kserve-84bf8f4d7f-htznc\" (UID: \"32af396a-976c-4f92-ab41-e0038cb69550\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"
Sep 08 14:09:37.878826 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.878803    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32af396a-976c-4f92-ab41-e0038cb69550-model-cache\") pod \"custom-route-timeout-test-kserve-84bf8f4d7f-htznc\" (UID: \"32af396a-976c-4f92-ab41-e0038cb69550\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"
Sep 08 14:09:37.880127 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.880105    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32af396a-976c-4f92-ab41-e0038cb69550-dshm\") pod \"custom-route-timeout-test-kserve-84bf8f4d7f-htznc\" (UID: \"32af396a-976c-4f92-ab41-e0038cb69550\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"
Sep 08 14:09:37.880496 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.880476    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32af396a-976c-4f92-ab41-e0038cb69550-tls-certs\") pod \"custom-route-timeout-test-kserve-84bf8f4d7f-htznc\" (UID: \"32af396a-976c-4f92-ab41-e0038cb69550\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"
Sep 08 14:09:37.887678 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:37.887653    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtmgm\" (UniqueName: \"kubernetes.io/projected/32af396a-976c-4f92-ab41-e0038cb69550-kube-api-access-rtmgm\") pod \"custom-route-timeout-test-kserve-84bf8f4d7f-htznc\" (UID: \"32af396a-976c-4f92-ab41-e0038cb69550\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"
Sep 08 14:09:38.038367 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:38.038267    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"
Sep 08 14:09:38.325124 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:38.325099    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"]
Sep 08 14:09:38.326572 ip-10-0-128-147 kubenswrapper[2558]: W0908 14:09:38.326539    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32af396a_976c_4f92_ab41_e0038cb69550.slice/crio-4e8de147c9526d363decbd1285fef412f1bfc8d021d1bc09e077384021ea2aa5 WatchSource:0}: Error finding container 4e8de147c9526d363decbd1285fef412f1bfc8d021d1bc09e077384021ea2aa5: Status 404 returned error can't find the container with id 4e8de147c9526d363decbd1285fef412f1bfc8d021d1bc09e077384021ea2aa5
Sep 08 14:09:38.833039 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:38.832996    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc" event={"ID":"32af396a-976c-4f92-ab41-e0038cb69550","Type":"ContainerStarted","Data":"7c435b0f336b00964d8fcba893e607ff728ccf69f13b8f6f48d63719005670a7"}
Sep 08 14:09:38.833497 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:38.833063    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc" event={"ID":"32af396a-976c-4f92-ab41-e0038cb69550","Type":"ContainerStarted","Data":"4e8de147c9526d363decbd1285fef412f1bfc8d021d1bc09e077384021ea2aa5"}
Sep 08 14:09:40.844326 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:40.844288    2558 generic.go:356] "Generic (PLEG): container finished" podID="32af396a-976c-4f92-ab41-e0038cb69550" containerID="7c435b0f336b00964d8fcba893e607ff728ccf69f13b8f6f48d63719005670a7" exitCode=0
Sep 08 14:09:40.844742 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:40.844370    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc" event={"ID":"32af396a-976c-4f92-ab41-e0038cb69550","Type":"ContainerDied","Data":"7c435b0f336b00964d8fcba893e607ff728ccf69f13b8f6f48d63719005670a7"}
Sep 08 14:09:41.852399 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:41.852358    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc" event={"ID":"32af396a-976c-4f92-ab41-e0038cb69550","Type":"ContainerStarted","Data":"86274de3b053eba1c2f58991580eebe3093bb3889c5cc2cc96a27b2a8251c094"}
Sep 08 14:09:41.880707 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:41.880642    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc" podStartSLOduration=2.619185074 podStartE2EDuration="4.880615102s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.261430028s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:09:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:09:41.879056651 +0000 UTC m=+3245.330817581" watchObservedRunningTime="2026-09-08 14:09:41.880615102 +0000 UTC m=+3245.332376034"
Sep 08 14:09:42.039332 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:42.039293    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"
Sep 08 14:09:48.039053 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:48.039012    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"
Sep 08 14:09:48.040546 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:48.040517    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc" podUID="32af396a-976c-4f92-ab41-e0038cb69550" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 08 14:09:58.039256 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:09:58.039210    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc" podUID="32af396a-976c-4f92-ab41-e0038cb69550" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 08 14:10:08.039021 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:10:08.038971    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc" podUID="32af396a-976c-4f92-ab41-e0038cb69550" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 08 14:10:18.039083 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:10:18.039037    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc" podUID="32af396a-976c-4f92-ab41-e0038cb69550" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 08 14:10:28.038797 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:10:28.038743    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc" podUID="32af396a-976c-4f92-ab41-e0038cb69550" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 08 14:10:38.038781 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:10:38.038736    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc" podUID="32af396a-976c-4f92-ab41-e0038cb69550" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 08 14:10:48.038881 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:10:48.038835    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc" podUID="32af396a-976c-4f92-ab41-e0038cb69550" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 08 14:10:58.039264 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:10:58.039217    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc" podUID="32af396a-976c-4f92-ab41-e0038cb69550" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 08 14:11:08.039330 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:08.039229    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc" podUID="32af396a-976c-4f92-ab41-e0038cb69550" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 08 14:11:18.039644 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:18.039571    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc" podUID="32af396a-976c-4f92-ab41-e0038cb69550" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 08 14:11:28.039793 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:28.039737    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc" podUID="32af396a-976c-4f92-ab41-e0038cb69550" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 08 14:11:38.049493 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:38.049451    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"
Sep 08 14:11:38.058066 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:38.058035    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"
Sep 08 14:11:42.532793 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:42.532754    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"]
Sep 08 14:11:57.561478 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:57.561435    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc" podUID="32af396a-976c-4f92-ab41-e0038cb69550" containerName="main" containerID="cri-o://86274de3b053eba1c2f58991580eebe3093bb3889c5cc2cc96a27b2a8251c094" gracePeriod=45
Sep 08 14:11:58.049974 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.049932    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc" podUID="32af396a-976c-4f92-ab41-e0038cb69550" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": dial tcp 10.132.0.70:8000: connect: connection refused"
Sep 08 14:11:58.311295 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.311238    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"
Sep 08 14:11:58.368693 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.368661    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtmgm\" (UniqueName: \"kubernetes.io/projected/32af396a-976c-4f92-ab41-e0038cb69550-kube-api-access-rtmgm\") pod \"32af396a-976c-4f92-ab41-e0038cb69550\" (UID: \"32af396a-976c-4f92-ab41-e0038cb69550\") "
Sep 08 14:11:58.368845 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.368698    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32af396a-976c-4f92-ab41-e0038cb69550-home\") pod \"32af396a-976c-4f92-ab41-e0038cb69550\" (UID: \"32af396a-976c-4f92-ab41-e0038cb69550\") "
Sep 08 14:11:58.368845 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.368750    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32af396a-976c-4f92-ab41-e0038cb69550-tls-certs\") pod \"32af396a-976c-4f92-ab41-e0038cb69550\" (UID: \"32af396a-976c-4f92-ab41-e0038cb69550\") "
Sep 08 14:11:58.368845 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.368778    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32af396a-976c-4f92-ab41-e0038cb69550-model-cache\") pod \"32af396a-976c-4f92-ab41-e0038cb69550\" (UID: \"32af396a-976c-4f92-ab41-e0038cb69550\") "
Sep 08 14:11:58.368845 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.368803    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32af396a-976c-4f92-ab41-e0038cb69550-tmp-dir\") pod \"32af396a-976c-4f92-ab41-e0038cb69550\" (UID: \"32af396a-976c-4f92-ab41-e0038cb69550\") "
Sep 08 14:11:58.368845 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.368835    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32af396a-976c-4f92-ab41-e0038cb69550-cabundle-cert\") pod \"32af396a-976c-4f92-ab41-e0038cb69550\" (UID: \"32af396a-976c-4f92-ab41-e0038cb69550\") "
Sep 08 14:11:58.369124 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.368878    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32af396a-976c-4f92-ab41-e0038cb69550-kserve-provision-location\") pod \"32af396a-976c-4f92-ab41-e0038cb69550\" (UID: \"32af396a-976c-4f92-ab41-e0038cb69550\") "
Sep 08 14:11:58.369124 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.368931    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32af396a-976c-4f92-ab41-e0038cb69550-dshm\") pod \"32af396a-976c-4f92-ab41-e0038cb69550\" (UID: \"32af396a-976c-4f92-ab41-e0038cb69550\") "
Sep 08 14:11:58.369238 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.369118    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32af396a-976c-4f92-ab41-e0038cb69550-model-cache" (OuterVolumeSpecName: "model-cache") pod "32af396a-976c-4f92-ab41-e0038cb69550" (UID: "32af396a-976c-4f92-ab41-e0038cb69550"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:11:58.369293 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.369271    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32af396a-976c-4f92-ab41-e0038cb69550-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:11:58.369365 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.369338    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32af396a-976c-4f92-ab41-e0038cb69550-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "32af396a-976c-4f92-ab41-e0038cb69550" (UID: "32af396a-976c-4f92-ab41-e0038cb69550"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:11:58.369468 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.369443    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32af396a-976c-4f92-ab41-e0038cb69550-home" (OuterVolumeSpecName: "home") pod "32af396a-976c-4f92-ab41-e0038cb69550" (UID: "32af396a-976c-4f92-ab41-e0038cb69550"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:11:58.371069 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.371036    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32af396a-976c-4f92-ab41-e0038cb69550-kube-api-access-rtmgm" (OuterVolumeSpecName: "kube-api-access-rtmgm") pod "32af396a-976c-4f92-ab41-e0038cb69550" (UID: "32af396a-976c-4f92-ab41-e0038cb69550"). InnerVolumeSpecName "kube-api-access-rtmgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:11:58.371186 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.371132    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32af396a-976c-4f92-ab41-e0038cb69550-dshm" (OuterVolumeSpecName: "dshm") pod "32af396a-976c-4f92-ab41-e0038cb69550" (UID: "32af396a-976c-4f92-ab41-e0038cb69550"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:11:58.371607 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.371573    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32af396a-976c-4f92-ab41-e0038cb69550-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "32af396a-976c-4f92-ab41-e0038cb69550" (UID: "32af396a-976c-4f92-ab41-e0038cb69550"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:11:58.380652 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.380620    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32af396a-976c-4f92-ab41-e0038cb69550-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "32af396a-976c-4f92-ab41-e0038cb69550" (UID: "32af396a-976c-4f92-ab41-e0038cb69550"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:11:58.391488 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.391456    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32af396a-976c-4f92-ab41-e0038cb69550-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "32af396a-976c-4f92-ab41-e0038cb69550" (UID: "32af396a-976c-4f92-ab41-e0038cb69550"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:11:58.470470 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.470439    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtmgm\" (UniqueName: \"kubernetes.io/projected/32af396a-976c-4f92-ab41-e0038cb69550-kube-api-access-rtmgm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:11:58.470470 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.470467    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32af396a-976c-4f92-ab41-e0038cb69550-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:11:58.470678 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.470477    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32af396a-976c-4f92-ab41-e0038cb69550-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:11:58.470678 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.470488    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32af396a-976c-4f92-ab41-e0038cb69550-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:11:58.470678 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.470496    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32af396a-976c-4f92-ab41-e0038cb69550-cabundle-cert\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:11:58.470678 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.470504    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32af396a-976c-4f92-ab41-e0038cb69550-kserve-provision-location\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:11:58.470678 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.470511    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32af396a-976c-4f92-ab41-e0038cb69550-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:11:58.546832 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.546791    2558 generic.go:356] "Generic (PLEG): container finished" podID="32af396a-976c-4f92-ab41-e0038cb69550" containerID="86274de3b053eba1c2f58991580eebe3093bb3889c5cc2cc96a27b2a8251c094" exitCode=0
Sep 08 14:11:58.547018 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.546858    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"
Sep 08 14:11:58.547018 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.546873    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc" event={"ID":"32af396a-976c-4f92-ab41-e0038cb69550","Type":"ContainerDied","Data":"86274de3b053eba1c2f58991580eebe3093bb3889c5cc2cc96a27b2a8251c094"}
Sep 08 14:11:58.547018 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.546969    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc" event={"ID":"32af396a-976c-4f92-ab41-e0038cb69550","Type":"ContainerDied","Data":"4e8de147c9526d363decbd1285fef412f1bfc8d021d1bc09e077384021ea2aa5"}
Sep 08 14:11:58.547018 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.547000    2558 scope.go:117] "RemoveContainer" containerID="86274de3b053eba1c2f58991580eebe3093bb3889c5cc2cc96a27b2a8251c094"
Sep 08 14:11:58.555973 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.555956    2558 scope.go:117] "RemoveContainer" containerID="7c435b0f336b00964d8fcba893e607ff728ccf69f13b8f6f48d63719005670a7"
Sep 08 14:11:58.564504 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.564311    2558 scope.go:117] "RemoveContainer" containerID="86274de3b053eba1c2f58991580eebe3093bb3889c5cc2cc96a27b2a8251c094"
Sep 08 14:11:58.564754 ip-10-0-128-147 kubenswrapper[2558]: E0908 14:11:58.564576    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86274de3b053eba1c2f58991580eebe3093bb3889c5cc2cc96a27b2a8251c094\": container with ID starting with 86274de3b053eba1c2f58991580eebe3093bb3889c5cc2cc96a27b2a8251c094 not found: ID does not exist" containerID="86274de3b053eba1c2f58991580eebe3093bb3889c5cc2cc96a27b2a8251c094"
Sep 08 14:11:58.564754 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.564623    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86274de3b053eba1c2f58991580eebe3093bb3889c5cc2cc96a27b2a8251c094"} err="failed to get container status \"86274de3b053eba1c2f58991580eebe3093bb3889c5cc2cc96a27b2a8251c094\": rpc error: code = NotFound desc = could not find container \"86274de3b053eba1c2f58991580eebe3093bb3889c5cc2cc96a27b2a8251c094\": container with ID starting with 86274de3b053eba1c2f58991580eebe3093bb3889c5cc2cc96a27b2a8251c094 not found: ID does not exist"
Sep 08 14:11:58.564754 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.564643    2558 scope.go:117] "RemoveContainer" containerID="7c435b0f336b00964d8fcba893e607ff728ccf69f13b8f6f48d63719005670a7"
Sep 08 14:11:58.564885 ip-10-0-128-147 kubenswrapper[2558]: E0908 14:11:58.564866    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c435b0f336b00964d8fcba893e607ff728ccf69f13b8f6f48d63719005670a7\": container with ID starting with 7c435b0f336b00964d8fcba893e607ff728ccf69f13b8f6f48d63719005670a7 not found: ID does not exist" containerID="7c435b0f336b00964d8fcba893e607ff728ccf69f13b8f6f48d63719005670a7"
Sep 08 14:11:58.564922 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.564889    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c435b0f336b00964d8fcba893e607ff728ccf69f13b8f6f48d63719005670a7"} err="failed to get container status \"7c435b0f336b00964d8fcba893e607ff728ccf69f13b8f6f48d63719005670a7\": rpc error: code = NotFound desc = could not find container \"7c435b0f336b00964d8fcba893e607ff728ccf69f13b8f6f48d63719005670a7\": container with ID starting with 7c435b0f336b00964d8fcba893e607ff728ccf69f13b8f6f48d63719005670a7 not found: ID does not exist"
Sep 08 14:11:58.568129 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.568103    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"]
Sep 08 14:11:58.574222 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:58.574196    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-84bf8f4d7f-htznc"]
Sep 08 14:11:59.053242 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:11:59.053208    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32af396a-976c-4f92-ab41-e0038cb69550" path="/var/lib/kubelet/pods/32af396a-976c-4f92-ab41-e0038cb69550/volumes"
Sep 08 14:12:22.729769 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.729728    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"]
Sep 08 14:12:22.730244 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.730103    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32af396a-976c-4f92-ab41-e0038cb69550" containerName="main"
Sep 08 14:12:22.730244 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.730117    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="32af396a-976c-4f92-ab41-e0038cb69550" containerName="main"
Sep 08 14:12:22.730244 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.730136    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32af396a-976c-4f92-ab41-e0038cb69550" containerName="storage-initializer"
Sep 08 14:12:22.730244 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.730142    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="32af396a-976c-4f92-ab41-e0038cb69550" containerName="storage-initializer"
Sep 08 14:12:22.730244 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.730224    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="32af396a-976c-4f92-ab41-e0038cb69550" containerName="main"
Sep 08 14:12:22.733498 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.733472    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"
Sep 08 14:12:22.736363 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.736335    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 08 14:12:22.736530 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.736402    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 08 14:12:22.736530 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.736524    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 14:12:22.736754 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.736402    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 08 14:12:22.736754 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.736502    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 08 14:12:22.736754 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.736626    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-jmqm4\""
Sep 08 14:12:22.745857 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.745831    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"]
Sep 08 14:12:22.784361 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.784326    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7746a427-d2ea-405c-acc6-d5477043e172-kserve-provision-location\") pod \"stop-feature-test-kserve-895d8956b-5sjkg\" (UID: \"7746a427-d2ea-405c-acc6-d5477043e172\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"
Sep 08 14:12:22.784361 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.784364    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7746a427-d2ea-405c-acc6-d5477043e172-model-cache\") pod \"stop-feature-test-kserve-895d8956b-5sjkg\" (UID: \"7746a427-d2ea-405c-acc6-d5477043e172\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"
Sep 08 14:12:22.784578 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.784400    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxg8v\" (UniqueName: \"kubernetes.io/projected/7746a427-d2ea-405c-acc6-d5477043e172-kube-api-access-vxg8v\") pod \"stop-feature-test-kserve-895d8956b-5sjkg\" (UID: \"7746a427-d2ea-405c-acc6-d5477043e172\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"
Sep 08 14:12:22.784578 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.784461    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7746a427-d2ea-405c-acc6-d5477043e172-dshm\") pod \"stop-feature-test-kserve-895d8956b-5sjkg\" (UID: \"7746a427-d2ea-405c-acc6-d5477043e172\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"
Sep 08 14:12:22.784578 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.784531    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7746a427-d2ea-405c-acc6-d5477043e172-cabundle-cert\") pod \"stop-feature-test-kserve-895d8956b-5sjkg\" (UID: \"7746a427-d2ea-405c-acc6-d5477043e172\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"
Sep 08 14:12:22.784578 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.784563    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7746a427-d2ea-405c-acc6-d5477043e172-home\") pod \"stop-feature-test-kserve-895d8956b-5sjkg\" (UID: \"7746a427-d2ea-405c-acc6-d5477043e172\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"
Sep 08 14:12:22.784744 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.784584    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7746a427-d2ea-405c-acc6-d5477043e172-tmp-dir\") pod \"stop-feature-test-kserve-895d8956b-5sjkg\" (UID: \"7746a427-d2ea-405c-acc6-d5477043e172\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"
Sep 08 14:12:22.784744 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.784669    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7746a427-d2ea-405c-acc6-d5477043e172-tls-certs\") pod \"stop-feature-test-kserve-895d8956b-5sjkg\" (UID: \"7746a427-d2ea-405c-acc6-d5477043e172\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"
Sep 08 14:12:22.885084 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.885032    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxg8v\" (UniqueName: \"kubernetes.io/projected/7746a427-d2ea-405c-acc6-d5477043e172-kube-api-access-vxg8v\") pod \"stop-feature-test-kserve-895d8956b-5sjkg\" (UID: \"7746a427-d2ea-405c-acc6-d5477043e172\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"
Sep 08 14:12:22.885084 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.885089    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7746a427-d2ea-405c-acc6-d5477043e172-dshm\") pod \"stop-feature-test-kserve-895d8956b-5sjkg\" (UID: \"7746a427-d2ea-405c-acc6-d5477043e172\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"
Sep 08 14:12:22.885307 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.885134    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7746a427-d2ea-405c-acc6-d5477043e172-cabundle-cert\") pod \"stop-feature-test-kserve-895d8956b-5sjkg\" (UID: \"7746a427-d2ea-405c-acc6-d5477043e172\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"
Sep 08 14:12:22.885307 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.885159    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7746a427-d2ea-405c-acc6-d5477043e172-home\") pod \"stop-feature-test-kserve-895d8956b-5sjkg\" (UID: \"7746a427-d2ea-405c-acc6-d5477043e172\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"
Sep 08 14:12:22.885307 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.885181    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7746a427-d2ea-405c-acc6-d5477043e172-tmp-dir\") pod \"stop-feature-test-kserve-895d8956b-5sjkg\" (UID: \"7746a427-d2ea-405c-acc6-d5477043e172\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"
Sep 08 14:12:22.885464 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.885334    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7746a427-d2ea-405c-acc6-d5477043e172-tls-certs\") pod \"stop-feature-test-kserve-895d8956b-5sjkg\" (UID: \"7746a427-d2ea-405c-acc6-d5477043e172\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"
Sep 08 14:12:22.885464 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.885417    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7746a427-d2ea-405c-acc6-d5477043e172-kserve-provision-location\") pod \"stop-feature-test-kserve-895d8956b-5sjkg\" (UID: \"7746a427-d2ea-405c-acc6-d5477043e172\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"
Sep 08 14:12:22.885464 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.885459    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7746a427-d2ea-405c-acc6-d5477043e172-model-cache\") pod \"stop-feature-test-kserve-895d8956b-5sjkg\" (UID: \"7746a427-d2ea-405c-acc6-d5477043e172\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"
Sep 08 14:12:22.885729 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.885699    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7746a427-d2ea-405c-acc6-d5477043e172-tmp-dir\") pod \"stop-feature-test-kserve-895d8956b-5sjkg\" (UID: \"7746a427-d2ea-405c-acc6-d5477043e172\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"
Sep 08 14:12:22.885881 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.885745    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7746a427-d2ea-405c-acc6-d5477043e172-home\") pod \"stop-feature-test-kserve-895d8956b-5sjkg\" (UID: \"7746a427-d2ea-405c-acc6-d5477043e172\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"
Sep 08 14:12:22.885881 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.885824    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7746a427-d2ea-405c-acc6-d5477043e172-kserve-provision-location\") pod \"stop-feature-test-kserve-895d8956b-5sjkg\" (UID: \"7746a427-d2ea-405c-acc6-d5477043e172\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"
Sep 08 14:12:22.886066 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.885903    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7746a427-d2ea-405c-acc6-d5477043e172-model-cache\") pod \"stop-feature-test-kserve-895d8956b-5sjkg\" (UID: \"7746a427-d2ea-405c-acc6-d5477043e172\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"
Sep 08 14:12:22.886066 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.885972    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7746a427-d2ea-405c-acc6-d5477043e172-cabundle-cert\") pod \"stop-feature-test-kserve-895d8956b-5sjkg\" (UID: \"7746a427-d2ea-405c-acc6-d5477043e172\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"
Sep 08 14:12:22.887548 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.887528    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7746a427-d2ea-405c-acc6-d5477043e172-dshm\") pod \"stop-feature-test-kserve-895d8956b-5sjkg\" (UID: \"7746a427-d2ea-405c-acc6-d5477043e172\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"
Sep 08 14:12:22.887831 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.887816    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7746a427-d2ea-405c-acc6-d5477043e172-tls-certs\") pod \"stop-feature-test-kserve-895d8956b-5sjkg\" (UID: \"7746a427-d2ea-405c-acc6-d5477043e172\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"
Sep 08 14:12:22.898551 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:22.898508    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxg8v\" (UniqueName: \"kubernetes.io/projected/7746a427-d2ea-405c-acc6-d5477043e172-kube-api-access-vxg8v\") pod \"stop-feature-test-kserve-895d8956b-5sjkg\" (UID: \"7746a427-d2ea-405c-acc6-d5477043e172\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"
Sep 08 14:12:23.045464 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:23.045355    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"
Sep 08 14:12:23.341740 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:23.341710    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"]
Sep 08 14:12:23.343501 ip-10-0-128-147 kubenswrapper[2558]: W0908 14:12:23.343463    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7746a427_d2ea_405c_acc6_d5477043e172.slice/crio-0bb639da3dbae853935e53a8b69ca6a2f57674c4c7fe907921325eb06aa87c36 WatchSource:0}: Error finding container 0bb639da3dbae853935e53a8b69ca6a2f57674c4c7fe907921325eb06aa87c36: Status 404 returned error can't find the container with id 0bb639da3dbae853935e53a8b69ca6a2f57674c4c7fe907921325eb06aa87c36
Sep 08 14:12:23.668630 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:23.668573    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg" event={"ID":"7746a427-d2ea-405c-acc6-d5477043e172","Type":"ContainerStarted","Data":"03ad24ac9f18d8aa57324aa6c85e5ae4a0b0c8dfc8098fa0f33eb0fe1534bed6"}
Sep 08 14:12:23.668840 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:23.668679    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg" event={"ID":"7746a427-d2ea-405c-acc6-d5477043e172","Type":"ContainerStarted","Data":"0bb639da3dbae853935e53a8b69ca6a2f57674c4c7fe907921325eb06aa87c36"}
Sep 08 14:12:25.681481 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:25.681449    2558 generic.go:356] "Generic (PLEG): container finished" podID="7746a427-d2ea-405c-acc6-d5477043e172" containerID="03ad24ac9f18d8aa57324aa6c85e5ae4a0b0c8dfc8098fa0f33eb0fe1534bed6" exitCode=0
Sep 08 14:12:25.681895 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:25.681520    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg" event={"ID":"7746a427-d2ea-405c-acc6-d5477043e172","Type":"ContainerDied","Data":"03ad24ac9f18d8aa57324aa6c85e5ae4a0b0c8dfc8098fa0f33eb0fe1534bed6"}
Sep 08 14:12:26.692342 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:26.692297    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg" event={"ID":"7746a427-d2ea-405c-acc6-d5477043e172","Type":"ContainerStarted","Data":"1d737e7c2a703530cb873ed236a60d918e3f0799ce21b946c1b44f7516707696"}
Sep 08 14:12:26.719096 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:26.719037    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg" podStartSLOduration=2.485127076 podStartE2EDuration="4.719019538s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.233892462s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:12:26.717135922 +0000 UTC m=+3410.168896854" watchObservedRunningTime="2026-09-08 14:12:26.719019538 +0000 UTC m=+3410.170780468"
Sep 08 14:12:27.046132 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:27.046034    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"
Sep 08 14:12:33.046414 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:33.046325    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"
Sep 08 14:12:33.047876 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:33.047847    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg" podUID="7746a427-d2ea-405c-acc6-d5477043e172" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 08 14:12:43.046242 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:43.046183    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg" podUID="7746a427-d2ea-405c-acc6-d5477043e172" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 08 14:12:47.347988 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.347952    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"]
Sep 08 14:12:47.352495 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.352475    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"
Sep 08 14:12:47.354950 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.354926    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 14:12:47.355085 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.354953    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 08 14:12:47.355159 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.354965    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Sep 08 14:12:47.355237 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.354986    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 08 14:12:47.355237 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.355027    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-4r66n\""
Sep 08 14:12:47.355359 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.355032    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 08 14:12:47.391966 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.391932    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"]
Sep 08 14:12:47.518249 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.518207    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-tmp-dir\") pod \"router-with-refs-test-kserve-6fff8966d7-vmzrf\" (UID: \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"
Sep 08 14:12:47.518249 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.518259    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-dshm\") pod \"router-with-refs-test-kserve-6fff8966d7-vmzrf\" (UID: \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"
Sep 08 14:12:47.518487 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.518279    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-cabundle-cert\") pod \"router-with-refs-test-kserve-6fff8966d7-vmzrf\" (UID: \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"
Sep 08 14:12:47.518487 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.518402    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-tls-certs\") pod \"router-with-refs-test-kserve-6fff8966d7-vmzrf\" (UID: \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"
Sep 08 14:12:47.518487 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.518441    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-model-cache\") pod \"router-with-refs-test-kserve-6fff8966d7-vmzrf\" (UID: \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"
Sep 08 14:12:47.518487 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.518473    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-kserve-provision-location\") pod \"router-with-refs-test-kserve-6fff8966d7-vmzrf\" (UID: \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"
Sep 08 14:12:47.518662 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.518507    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-642mh\" (UniqueName: \"kubernetes.io/projected/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-kube-api-access-642mh\") pod \"router-with-refs-test-kserve-6fff8966d7-vmzrf\" (UID: \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"
Sep 08 14:12:47.518662 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.518527    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-home\") pod \"router-with-refs-test-kserve-6fff8966d7-vmzrf\" (UID: \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"
Sep 08 14:12:47.619424 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.619393    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-kserve-provision-location\") pod \"router-with-refs-test-kserve-6fff8966d7-vmzrf\" (UID: \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"
Sep 08 14:12:47.619564 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.619434    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-642mh\" (UniqueName: \"kubernetes.io/projected/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-kube-api-access-642mh\") pod \"router-with-refs-test-kserve-6fff8966d7-vmzrf\" (UID: \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"
Sep 08 14:12:47.619564 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.619456    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-home\") pod \"router-with-refs-test-kserve-6fff8966d7-vmzrf\" (UID: \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"
Sep 08 14:12:47.619564 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.619493    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-tmp-dir\") pod \"router-with-refs-test-kserve-6fff8966d7-vmzrf\" (UID: \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"
Sep 08 14:12:47.619564 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.619515    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-dshm\") pod \"router-with-refs-test-kserve-6fff8966d7-vmzrf\" (UID: \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"
Sep 08 14:12:47.619564 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.619533    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-cabundle-cert\") pod \"router-with-refs-test-kserve-6fff8966d7-vmzrf\" (UID: \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"
Sep 08 14:12:47.619880 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.619615    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-tls-certs\") pod \"router-with-refs-test-kserve-6fff8966d7-vmzrf\" (UID: \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"
Sep 08 14:12:47.619880 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.619648    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-model-cache\") pod \"router-with-refs-test-kserve-6fff8966d7-vmzrf\" (UID: \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"
Sep 08 14:12:47.619993 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.619911    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-kserve-provision-location\") pod \"router-with-refs-test-kserve-6fff8966d7-vmzrf\" (UID: \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"
Sep 08 14:12:47.619993 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.619972    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-home\") pod \"router-with-refs-test-kserve-6fff8966d7-vmzrf\" (UID: \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"
Sep 08 14:12:47.620100 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.620038    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-model-cache\") pod \"router-with-refs-test-kserve-6fff8966d7-vmzrf\" (UID: \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"
Sep 08 14:12:47.620100 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.620060    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-tmp-dir\") pod \"router-with-refs-test-kserve-6fff8966d7-vmzrf\" (UID: \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"
Sep 08 14:12:47.620332 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.620313    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-cabundle-cert\") pod \"router-with-refs-test-kserve-6fff8966d7-vmzrf\" (UID: \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"
Sep 08 14:12:47.621931 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.621910    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-dshm\") pod \"router-with-refs-test-kserve-6fff8966d7-vmzrf\" (UID: \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"
Sep 08 14:12:47.622256 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.622234    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-tls-certs\") pod \"router-with-refs-test-kserve-6fff8966d7-vmzrf\" (UID: \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"
Sep 08 14:12:47.627712 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.627694    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-642mh\" (UniqueName: \"kubernetes.io/projected/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-kube-api-access-642mh\") pod \"router-with-refs-test-kserve-6fff8966d7-vmzrf\" (UID: \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"
Sep 08 14:12:47.666582 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.666555    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"
Sep 08 14:12:47.954425 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:47.954346    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"]
Sep 08 14:12:47.958058 ip-10-0-128-147 kubenswrapper[2558]: W0908 14:12:47.958029    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b5eac6_9f7d_42ca_b1f7_a469e7b84b9f.slice/crio-cf7a522fadea6be051eb47737012096706e53c0664795e983a09de80f3417f68 WatchSource:0}: Error finding container cf7a522fadea6be051eb47737012096706e53c0664795e983a09de80f3417f68: Status 404 returned error can't find the container with id cf7a522fadea6be051eb47737012096706e53c0664795e983a09de80f3417f68
Sep 08 14:12:48.815538 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:48.815501    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf" event={"ID":"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f","Type":"ContainerStarted","Data":"60232ef0860d9d181ff2896a008516157d9a1e9eab58a98daabaf717deeba922"}
Sep 08 14:12:48.816059 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:48.815571    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf" event={"ID":"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f","Type":"ContainerStarted","Data":"cf7a522fadea6be051eb47737012096706e53c0664795e983a09de80f3417f68"}
Sep 08 14:12:50.827633 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:50.827571    2558 generic.go:356] "Generic (PLEG): container finished" podID="a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f" containerID="60232ef0860d9d181ff2896a008516157d9a1e9eab58a98daabaf717deeba922" exitCode=0
Sep 08 14:12:50.828152 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:50.827646    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf" event={"ID":"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f","Type":"ContainerDied","Data":"60232ef0860d9d181ff2896a008516157d9a1e9eab58a98daabaf717deeba922"}
Sep 08 14:12:51.835903 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:51.835868    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf" event={"ID":"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f","Type":"ContainerStarted","Data":"4025976e24760221f382d583e11318828ad383d5a146c399ca398d2096d87cfb"}
Sep 08 14:12:51.858702 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:51.858637    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf" podStartSLOduration=2.5555411489999997 podStartE2EDuration="4.858619856s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.303078707s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:12:51.857642177 +0000 UTC m=+3435.309403110" watchObservedRunningTime="2026-09-08 14:12:51.858619856 +0000 UTC m=+3435.310380790"
Sep 08 14:12:52.667493 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:52.667447    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"
Sep 08 14:12:53.046114 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:53.046023    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg" podUID="7746a427-d2ea-405c-acc6-d5477043e172" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 08 14:12:57.667666 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:57.667623    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"
Sep 08 14:12:57.669193 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:12:57.669156    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf" podUID="a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 08 14:13:03.046500 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:13:03.046444    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg" podUID="7746a427-d2ea-405c-acc6-d5477043e172" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 08 14:13:07.667843 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:13:07.667793    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf" podUID="a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 08 14:13:13.046434 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:13:13.046385    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg" podUID="7746a427-d2ea-405c-acc6-d5477043e172" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 08 14:13:17.667438 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:13:17.667382    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf" podUID="a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 08 14:13:23.046253 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:13:23.046202    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg" podUID="7746a427-d2ea-405c-acc6-d5477043e172" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 08 14:13:27.667731 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:13:27.667658    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf" podUID="a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 08 14:13:33.046349 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:13:33.046290    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg" podUID="7746a427-d2ea-405c-acc6-d5477043e172" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 08 14:13:37.668009 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:13:37.667967    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf" podUID="a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 08 14:13:43.046137 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:13:43.046089    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg" podUID="7746a427-d2ea-405c-acc6-d5477043e172" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 08 14:13:47.667184 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:13:47.667141    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf" podUID="a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 08 14:13:53.045875 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:13:53.045828    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg" podUID="7746a427-d2ea-405c-acc6-d5477043e172" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 08 14:13:57.668065 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:13:57.668014    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf" podUID="a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 08 14:14:03.046914 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:14:03.046814    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg" podUID="7746a427-d2ea-405c-acc6-d5477043e172" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 08 14:14:07.667673 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:14:07.667631    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf" podUID="a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 08 14:14:13.046676 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:14:13.046618    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg" podUID="7746a427-d2ea-405c-acc6-d5477043e172" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 08 14:14:17.667192 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:14:17.667132    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf" podUID="a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 08 14:14:23.046214 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:14:23.046155    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg" podUID="7746a427-d2ea-405c-acc6-d5477043e172" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 08 14:14:27.667340 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:14:27.667287    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf" podUID="a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 08 14:14:33.046271 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:14:33.046218    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg" podUID="7746a427-d2ea-405c-acc6-d5477043e172" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 08 14:14:37.667110 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:14:37.667064    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf" podUID="a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 08 14:14:43.057945 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:14:43.057905    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"
Sep 08 14:14:43.065738 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:14:43.065706    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"
Sep 08 14:14:44.848216 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:14:44.848180    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"]
Sep 08 14:14:47.667894 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:14:47.667842    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf" podUID="a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 08 14:14:57.667564 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:14:57.667505    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf" podUID="a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 08 14:14:59.872816 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:14:59.872769    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg" podUID="7746a427-d2ea-405c-acc6-d5477043e172" containerName="main" containerID="cri-o://1d737e7c2a703530cb873ed236a60d918e3f0799ce21b946c1b44f7516707696" gracePeriod=45
Sep 08 14:15:00.059159 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:00.059110    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg" podUID="7746a427-d2ea-405c-acc6-d5477043e172" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 08 14:15:00.565418 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:00.565380    2558 generic.go:356] "Generic (PLEG): container finished" podID="7746a427-d2ea-405c-acc6-d5477043e172" containerID="1d737e7c2a703530cb873ed236a60d918e3f0799ce21b946c1b44f7516707696" exitCode=0
Sep 08 14:15:00.565623 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:00.565455    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg" event={"ID":"7746a427-d2ea-405c-acc6-d5477043e172","Type":"ContainerDied","Data":"1d737e7c2a703530cb873ed236a60d918e3f0799ce21b946c1b44f7516707696"}
Sep 08 14:15:00.632247 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:00.632212    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"
Sep 08 14:15:00.749389 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:00.749284    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7746a427-d2ea-405c-acc6-d5477043e172-model-cache\") pod \"7746a427-d2ea-405c-acc6-d5477043e172\" (UID: \"7746a427-d2ea-405c-acc6-d5477043e172\") "
Sep 08 14:15:00.749576 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:00.749410    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7746a427-d2ea-405c-acc6-d5477043e172-tmp-dir\") pod \"7746a427-d2ea-405c-acc6-d5477043e172\" (UID: \"7746a427-d2ea-405c-acc6-d5477043e172\") "
Sep 08 14:15:00.749576 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:00.749441    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7746a427-d2ea-405c-acc6-d5477043e172-dshm\") pod \"7746a427-d2ea-405c-acc6-d5477043e172\" (UID: \"7746a427-d2ea-405c-acc6-d5477043e172\") "
Sep 08 14:15:00.749576 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:00.749463    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7746a427-d2ea-405c-acc6-d5477043e172-cabundle-cert\") pod \"7746a427-d2ea-405c-acc6-d5477043e172\" (UID: \"7746a427-d2ea-405c-acc6-d5477043e172\") "
Sep 08 14:15:00.749576 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:00.749489    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxg8v\" (UniqueName: \"kubernetes.io/projected/7746a427-d2ea-405c-acc6-d5477043e172-kube-api-access-vxg8v\") pod \"7746a427-d2ea-405c-acc6-d5477043e172\" (UID: \"7746a427-d2ea-405c-acc6-d5477043e172\") "
Sep 08 14:15:00.749576 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:00.749511    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7746a427-d2ea-405c-acc6-d5477043e172-home\") pod \"7746a427-d2ea-405c-acc6-d5477043e172\" (UID: \"7746a427-d2ea-405c-acc6-d5477043e172\") "
Sep 08 14:15:00.749576 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:00.749542    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7746a427-d2ea-405c-acc6-d5477043e172-kserve-provision-location\") pod \"7746a427-d2ea-405c-acc6-d5477043e172\" (UID: \"7746a427-d2ea-405c-acc6-d5477043e172\") "
Sep 08 14:15:00.749943 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:00.749580    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7746a427-d2ea-405c-acc6-d5477043e172-model-cache" (OuterVolumeSpecName: "model-cache") pod "7746a427-d2ea-405c-acc6-d5477043e172" (UID: "7746a427-d2ea-405c-acc6-d5477043e172"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:15:00.749943 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:00.749626    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7746a427-d2ea-405c-acc6-d5477043e172-tls-certs\") pod \"7746a427-d2ea-405c-acc6-d5477043e172\" (UID: \"7746a427-d2ea-405c-acc6-d5477043e172\") "
Sep 08 14:15:00.749943 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:00.749892    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7746a427-d2ea-405c-acc6-d5477043e172-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7746a427-d2ea-405c-acc6-d5477043e172" (UID: "7746a427-d2ea-405c-acc6-d5477043e172"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:15:00.750116 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:00.749974    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7746a427-d2ea-405c-acc6-d5477043e172-cabundle-cert\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:15:00.750116 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:00.749993    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7746a427-d2ea-405c-acc6-d5477043e172-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:15:00.750624 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:00.750569    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7746a427-d2ea-405c-acc6-d5477043e172-home" (OuterVolumeSpecName: "home") pod "7746a427-d2ea-405c-acc6-d5477043e172" (UID: "7746a427-d2ea-405c-acc6-d5477043e172"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:15:00.752509 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:00.752482    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7746a427-d2ea-405c-acc6-d5477043e172-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7746a427-d2ea-405c-acc6-d5477043e172" (UID: "7746a427-d2ea-405c-acc6-d5477043e172"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:15:00.752736 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:00.752700    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7746a427-d2ea-405c-acc6-d5477043e172-dshm" (OuterVolumeSpecName: "dshm") pod "7746a427-d2ea-405c-acc6-d5477043e172" (UID: "7746a427-d2ea-405c-acc6-d5477043e172"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:15:00.752827 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:00.752796    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7746a427-d2ea-405c-acc6-d5477043e172-kube-api-access-vxg8v" (OuterVolumeSpecName: "kube-api-access-vxg8v") pod "7746a427-d2ea-405c-acc6-d5477043e172" (UID: "7746a427-d2ea-405c-acc6-d5477043e172"). InnerVolumeSpecName "kube-api-access-vxg8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:15:00.757641 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:00.757608    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7746a427-d2ea-405c-acc6-d5477043e172-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7746a427-d2ea-405c-acc6-d5477043e172" (UID: "7746a427-d2ea-405c-acc6-d5477043e172"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:15:00.767823 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:00.767787    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7746a427-d2ea-405c-acc6-d5477043e172-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7746a427-d2ea-405c-acc6-d5477043e172" (UID: "7746a427-d2ea-405c-acc6-d5477043e172"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:15:00.850677 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:00.850635    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7746a427-d2ea-405c-acc6-d5477043e172-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:15:00.850677 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:00.850665    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7746a427-d2ea-405c-acc6-d5477043e172-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:15:00.850677 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:00.850673    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxg8v\" (UniqueName: \"kubernetes.io/projected/7746a427-d2ea-405c-acc6-d5477043e172-kube-api-access-vxg8v\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:15:00.850677 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:00.850683    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7746a427-d2ea-405c-acc6-d5477043e172-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:15:00.850945 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:00.850691    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7746a427-d2ea-405c-acc6-d5477043e172-kserve-provision-location\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:15:00.850945 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:00.850701    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7746a427-d2ea-405c-acc6-d5477043e172-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:15:01.572173 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:01.572143    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"
Sep 08 14:15:01.572691 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:01.572175    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg" event={"ID":"7746a427-d2ea-405c-acc6-d5477043e172","Type":"ContainerDied","Data":"0bb639da3dbae853935e53a8b69ca6a2f57674c4c7fe907921325eb06aa87c36"}
Sep 08 14:15:01.572691 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:01.572240    2558 scope.go:117] "RemoveContainer" containerID="1d737e7c2a703530cb873ed236a60d918e3f0799ce21b946c1b44f7516707696"
Sep 08 14:15:01.581139 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:01.581116    2558 scope.go:117] "RemoveContainer" containerID="03ad24ac9f18d8aa57324aa6c85e5ae4a0b0c8dfc8098fa0f33eb0fe1534bed6"
Sep 08 14:15:01.590312 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:01.590288    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"]
Sep 08 14:15:01.596490 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:01.596463    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-895d8956b-5sjkg"]
Sep 08 14:15:03.053673 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:03.053639    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7746a427-d2ea-405c-acc6-d5477043e172" path="/var/lib/kubelet/pods/7746a427-d2ea-405c-acc6-d5477043e172/volumes"
Sep 08 14:15:07.678081 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:07.678048    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"
Sep 08 14:15:07.688049 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:07.688021    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"
Sep 08 14:15:12.274058 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:12.274022    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"]
Sep 08 14:15:27.297745 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:27.297699    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf" podUID="a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f" containerName="main" containerID="cri-o://4025976e24760221f382d583e11318828ad383d5a146c399ca398d2096d87cfb" gracePeriod=45
Sep 08 14:15:27.679302 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:27.679255    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf" podUID="a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Sep 08 14:15:28.077770 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.077741    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"
Sep 08 14:15:28.085653 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.085629    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-tls-certs\") pod \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\" (UID: \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\") "
Sep 08 14:15:28.085798 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.085728    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-kserve-provision-location\") pod \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\" (UID: \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\") "
Sep 08 14:15:28.085798 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.085753    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-cabundle-cert\") pod \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\" (UID: \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\") "
Sep 08 14:15:28.085798 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.085776    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-642mh\" (UniqueName: \"kubernetes.io/projected/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-kube-api-access-642mh\") pod \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\" (UID: \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\") "
Sep 08 14:15:28.086008 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.085816    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-dshm\") pod \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\" (UID: \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\") "
Sep 08 14:15:28.086008 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.085942    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-model-cache\") pod \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\" (UID: \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\") "
Sep 08 14:15:28.086008 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.085973    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-home\") pod \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\" (UID: \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\") "
Sep 08 14:15:28.086008 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.085999    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-tmp-dir\") pod \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\" (UID: \"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f\") "
Sep 08 14:15:28.086283 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.086253    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f" (UID: "a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:15:28.086768 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.086725    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-model-cache" (OuterVolumeSpecName: "model-cache") pod "a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f" (UID: "a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:15:28.086877 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.086779    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-home" (OuterVolumeSpecName: "home") pod "a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f" (UID: "a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:15:28.088080 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.088055    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f" (UID: "a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:15:28.088320 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.088303    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-dshm" (OuterVolumeSpecName: "dshm") pod "a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f" (UID: "a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:15:28.088394 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.088327    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-kube-api-access-642mh" (OuterVolumeSpecName: "kube-api-access-642mh") pod "a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f" (UID: "a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f"). InnerVolumeSpecName "kube-api-access-642mh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:15:28.106526 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.106485    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f" (UID: "a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:15:28.115043 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.114998    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f" (UID: "a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:15:28.187104 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.187065    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:15:28.187104 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.187097    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:15:28.187104 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.187107    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:15:28.187104 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.187117    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:15:28.187397 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.187125    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:15:28.187397 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.187133    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-kserve-provision-location\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:15:28.187397 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.187142    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-cabundle-cert\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:15:28.187397 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.187151    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-642mh\" (UniqueName: \"kubernetes.io/projected/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f-kube-api-access-642mh\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:15:28.706894 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.706859    2558 generic.go:356] "Generic (PLEG): container finished" podID="a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f" containerID="4025976e24760221f382d583e11318828ad383d5a146c399ca398d2096d87cfb" exitCode=0
Sep 08 14:15:28.707401 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.706913    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"
Sep 08 14:15:28.707401 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.706920    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf" event={"ID":"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f","Type":"ContainerDied","Data":"4025976e24760221f382d583e11318828ad383d5a146c399ca398d2096d87cfb"}
Sep 08 14:15:28.707401 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.706978    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf" event={"ID":"a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f","Type":"ContainerDied","Data":"cf7a522fadea6be051eb47737012096706e53c0664795e983a09de80f3417f68"}
Sep 08 14:15:28.707401 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.707002    2558 scope.go:117] "RemoveContainer" containerID="4025976e24760221f382d583e11318828ad383d5a146c399ca398d2096d87cfb"
Sep 08 14:15:28.716241 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.716222    2558 scope.go:117] "RemoveContainer" containerID="60232ef0860d9d181ff2896a008516157d9a1e9eab58a98daabaf717deeba922"
Sep 08 14:15:28.724295 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.724277    2558 scope.go:117] "RemoveContainer" containerID="4025976e24760221f382d583e11318828ad383d5a146c399ca398d2096d87cfb"
Sep 08 14:15:28.724578 ip-10-0-128-147 kubenswrapper[2558]: E0908 14:15:28.724557    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4025976e24760221f382d583e11318828ad383d5a146c399ca398d2096d87cfb\": container with ID starting with 4025976e24760221f382d583e11318828ad383d5a146c399ca398d2096d87cfb not found: ID does not exist" containerID="4025976e24760221f382d583e11318828ad383d5a146c399ca398d2096d87cfb"
Sep 08 14:15:28.724645 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.724587    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4025976e24760221f382d583e11318828ad383d5a146c399ca398d2096d87cfb"} err="failed to get container status \"4025976e24760221f382d583e11318828ad383d5a146c399ca398d2096d87cfb\": rpc error: code = NotFound desc = could not find container \"4025976e24760221f382d583e11318828ad383d5a146c399ca398d2096d87cfb\": container with ID starting with 4025976e24760221f382d583e11318828ad383d5a146c399ca398d2096d87cfb not found: ID does not exist"
Sep 08 14:15:28.724645 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.724622    2558 scope.go:117] "RemoveContainer" containerID="60232ef0860d9d181ff2896a008516157d9a1e9eab58a98daabaf717deeba922"
Sep 08 14:15:28.724866 ip-10-0-128-147 kubenswrapper[2558]: E0908 14:15:28.724849    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60232ef0860d9d181ff2896a008516157d9a1e9eab58a98daabaf717deeba922\": container with ID starting with 60232ef0860d9d181ff2896a008516157d9a1e9eab58a98daabaf717deeba922 not found: ID does not exist" containerID="60232ef0860d9d181ff2896a008516157d9a1e9eab58a98daabaf717deeba922"
Sep 08 14:15:28.724907 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.724871    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60232ef0860d9d181ff2896a008516157d9a1e9eab58a98daabaf717deeba922"} err="failed to get container status \"60232ef0860d9d181ff2896a008516157d9a1e9eab58a98daabaf717deeba922\": rpc error: code = NotFound desc = could not find container \"60232ef0860d9d181ff2896a008516157d9a1e9eab58a98daabaf717deeba922\": container with ID starting with 60232ef0860d9d181ff2896a008516157d9a1e9eab58a98daabaf717deeba922 not found: ID does not exist"
Sep 08 14:15:28.733451 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.733416    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"]
Sep 08 14:15:28.740602 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:28.740563    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6fff8966d7-vmzrf"]
Sep 08 14:15:29.054383 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:29.054298    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f" path="/var/lib/kubelet/pods/a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f/volumes"
Sep 08 14:15:52.811365 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:52.811332    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"]
Sep 08 14:15:52.811807 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:52.811731    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f" containerName="storage-initializer"
Sep 08 14:15:52.811807 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:52.811752    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f" containerName="storage-initializer"
Sep 08 14:15:52.811807 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:52.811764    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7746a427-d2ea-405c-acc6-d5477043e172" containerName="main"
Sep 08 14:15:52.811807 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:52.811769    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7746a427-d2ea-405c-acc6-d5477043e172" containerName="main"
Sep 08 14:15:52.811807 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:52.811783    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f" containerName="main"
Sep 08 14:15:52.811807 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:52.811788    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f" containerName="main"
Sep 08 14:15:52.811807 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:52.811797    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7746a427-d2ea-405c-acc6-d5477043e172" containerName="storage-initializer"
Sep 08 14:15:52.811807 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:52.811802    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7746a427-d2ea-405c-acc6-d5477043e172" containerName="storage-initializer"
Sep 08 14:15:52.812061 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:52.811859    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7746a427-d2ea-405c-acc6-d5477043e172" containerName="main"
Sep 08 14:15:52.812061 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:52.811868    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3b5eac6-9f7d-42ca-b1f7-a469e7b84b9f" containerName="main"
Sep 08 14:15:52.816312 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:52.816292    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"
Sep 08 14:15:52.819079 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:52.819054    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-6w5p5\""
Sep 08 14:15:52.819224 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:52.819078    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Sep 08 14:15:52.819224 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:52.819087    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Sep 08 14:15:52.819483 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:52.819466    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Sep 08 14:15:52.819483 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:52.819478    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 14:15:52.820793 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:52.820774    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Sep 08 14:15:52.831341 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:52.831313    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"]
Sep 08 14:15:52.899760 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:52.899714    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a2de755-8371-4584-917a-5478ff9b74f0-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-57b89d5f64-vtz4p\" (UID: \"5a2de755-8371-4584-917a-5478ff9b74f0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"
Sep 08 14:15:52.899966 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:52.899821    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a2de755-8371-4584-917a-5478ff9b74f0-tmp-dir\") pod \"lora-single-adapter-test-kserve-57b89d5f64-vtz4p\" (UID: \"5a2de755-8371-4584-917a-5478ff9b74f0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"
Sep 08 14:15:52.899966 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:52.899877    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a2de755-8371-4584-917a-5478ff9b74f0-tls-certs\") pod \"lora-single-adapter-test-kserve-57b89d5f64-vtz4p\" (UID: \"5a2de755-8371-4584-917a-5478ff9b74f0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"
Sep 08 14:15:52.899966 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:52.899900    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a2de755-8371-4584-917a-5478ff9b74f0-cabundle-cert\") pod \"lora-single-adapter-test-kserve-57b89d5f64-vtz4p\" (UID: \"5a2de755-8371-4584-917a-5478ff9b74f0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"
Sep 08 14:15:52.899966 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:52.899921    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a2de755-8371-4584-917a-5478ff9b74f0-dshm\") pod \"lora-single-adapter-test-kserve-57b89d5f64-vtz4p\" (UID: \"5a2de755-8371-4584-917a-5478ff9b74f0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"
Sep 08 14:15:52.900142 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:52.899994    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a2de755-8371-4584-917a-5478ff9b74f0-model-cache\") pod \"lora-single-adapter-test-kserve-57b89d5f64-vtz4p\" (UID: \"5a2de755-8371-4584-917a-5478ff9b74f0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"
Sep 08 14:15:52.900142 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:52.900054    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dzph\" (UniqueName: \"kubernetes.io/projected/5a2de755-8371-4584-917a-5478ff9b74f0-kube-api-access-2dzph\") pod \"lora-single-adapter-test-kserve-57b89d5f64-vtz4p\" (UID: \"5a2de755-8371-4584-917a-5478ff9b74f0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"
Sep 08 14:15:52.900142 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:52.900078    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a2de755-8371-4584-917a-5478ff9b74f0-home\") pod \"lora-single-adapter-test-kserve-57b89d5f64-vtz4p\" (UID: \"5a2de755-8371-4584-917a-5478ff9b74f0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"
Sep 08 14:15:53.001444 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:53.001399    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dzph\" (UniqueName: \"kubernetes.io/projected/5a2de755-8371-4584-917a-5478ff9b74f0-kube-api-access-2dzph\") pod \"lora-single-adapter-test-kserve-57b89d5f64-vtz4p\" (UID: \"5a2de755-8371-4584-917a-5478ff9b74f0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"
Sep 08 14:15:53.001697 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:53.001454    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a2de755-8371-4584-917a-5478ff9b74f0-home\") pod \"lora-single-adapter-test-kserve-57b89d5f64-vtz4p\" (UID: \"5a2de755-8371-4584-917a-5478ff9b74f0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"
Sep 08 14:15:53.001697 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:53.001498    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a2de755-8371-4584-917a-5478ff9b74f0-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-57b89d5f64-vtz4p\" (UID: \"5a2de755-8371-4584-917a-5478ff9b74f0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"
Sep 08 14:15:53.001697 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:53.001543    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a2de755-8371-4584-917a-5478ff9b74f0-tmp-dir\") pod \"lora-single-adapter-test-kserve-57b89d5f64-vtz4p\" (UID: \"5a2de755-8371-4584-917a-5478ff9b74f0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"
Sep 08 14:15:53.001697 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:53.001586    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a2de755-8371-4584-917a-5478ff9b74f0-tls-certs\") pod \"lora-single-adapter-test-kserve-57b89d5f64-vtz4p\" (UID: \"5a2de755-8371-4584-917a-5478ff9b74f0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"
Sep 08 14:15:53.001697 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:53.001642    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a2de755-8371-4584-917a-5478ff9b74f0-cabundle-cert\") pod \"lora-single-adapter-test-kserve-57b89d5f64-vtz4p\" (UID: \"5a2de755-8371-4584-917a-5478ff9b74f0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"
Sep 08 14:15:53.001697 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:53.001672    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a2de755-8371-4584-917a-5478ff9b74f0-dshm\") pod \"lora-single-adapter-test-kserve-57b89d5f64-vtz4p\" (UID: \"5a2de755-8371-4584-917a-5478ff9b74f0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"
Sep 08 14:15:53.002020 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:53.001714    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a2de755-8371-4584-917a-5478ff9b74f0-model-cache\") pod \"lora-single-adapter-test-kserve-57b89d5f64-vtz4p\" (UID: \"5a2de755-8371-4584-917a-5478ff9b74f0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"
Sep 08 14:15:53.002020 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:53.001917    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a2de755-8371-4584-917a-5478ff9b74f0-home\") pod \"lora-single-adapter-test-kserve-57b89d5f64-vtz4p\" (UID: \"5a2de755-8371-4584-917a-5478ff9b74f0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"
Sep 08 14:15:53.002020 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:53.002003    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a2de755-8371-4584-917a-5478ff9b74f0-tmp-dir\") pod \"lora-single-adapter-test-kserve-57b89d5f64-vtz4p\" (UID: \"5a2de755-8371-4584-917a-5478ff9b74f0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"
Sep 08 14:15:53.002175 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:53.002064    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a2de755-8371-4584-917a-5478ff9b74f0-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-57b89d5f64-vtz4p\" (UID: \"5a2de755-8371-4584-917a-5478ff9b74f0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"
Sep 08 14:15:53.002175 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:53.002125    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a2de755-8371-4584-917a-5478ff9b74f0-model-cache\") pod \"lora-single-adapter-test-kserve-57b89d5f64-vtz4p\" (UID: \"5a2de755-8371-4584-917a-5478ff9b74f0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"
Sep 08 14:15:53.002465 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:53.002439    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a2de755-8371-4584-917a-5478ff9b74f0-cabundle-cert\") pod \"lora-single-adapter-test-kserve-57b89d5f64-vtz4p\" (UID: \"5a2de755-8371-4584-917a-5478ff9b74f0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"
Sep 08 14:15:53.004125 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:53.004096    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a2de755-8371-4584-917a-5478ff9b74f0-dshm\") pod \"lora-single-adapter-test-kserve-57b89d5f64-vtz4p\" (UID: \"5a2de755-8371-4584-917a-5478ff9b74f0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"
Sep 08 14:15:53.004344 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:53.004326    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a2de755-8371-4584-917a-5478ff9b74f0-tls-certs\") pod \"lora-single-adapter-test-kserve-57b89d5f64-vtz4p\" (UID: \"5a2de755-8371-4584-917a-5478ff9b74f0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"
Sep 08 14:15:53.009979 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:53.009958    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dzph\" (UniqueName: \"kubernetes.io/projected/5a2de755-8371-4584-917a-5478ff9b74f0-kube-api-access-2dzph\") pod \"lora-single-adapter-test-kserve-57b89d5f64-vtz4p\" (UID: \"5a2de755-8371-4584-917a-5478ff9b74f0\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"
Sep 08 14:15:53.127755 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:53.127706    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"
Sep 08 14:15:53.439530 ip-10-0-128-147 kubenswrapper[2558]: W0908 14:15:53.439497    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a2de755_8371_4584_917a_5478ff9b74f0.slice/crio-3a3a2eef82f5121e06abc871b5bc375f7239b0e8d7a6da5cc0535d55e1afe23d WatchSource:0}: Error finding container 3a3a2eef82f5121e06abc871b5bc375f7239b0e8d7a6da5cc0535d55e1afe23d: Status 404 returned error can't find the container with id 3a3a2eef82f5121e06abc871b5bc375f7239b0e8d7a6da5cc0535d55e1afe23d
Sep 08 14:15:53.442643 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:53.442576    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"]
Sep 08 14:15:53.835237 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:53.835143    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p" event={"ID":"5a2de755-8371-4584-917a-5478ff9b74f0","Type":"ContainerStarted","Data":"b2368d983c68cb25f80e64e41a970cc1577f1e24c2e9b9a71b2fdf17ddfe4e36"}
Sep 08 14:15:53.835237 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:53.835194    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p" event={"ID":"5a2de755-8371-4584-917a-5478ff9b74f0","Type":"ContainerStarted","Data":"3a3a2eef82f5121e06abc871b5bc375f7239b0e8d7a6da5cc0535d55e1afe23d"}
Sep 08 14:15:56.851120 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:56.851021    2558 generic.go:356] "Generic (PLEG): container finished" podID="5a2de755-8371-4584-917a-5478ff9b74f0" containerID="b2368d983c68cb25f80e64e41a970cc1577f1e24c2e9b9a71b2fdf17ddfe4e36" exitCode=0
Sep 08 14:15:56.851120 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:56.851095    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p" event={"ID":"5a2de755-8371-4584-917a-5478ff9b74f0","Type":"ContainerDied","Data":"b2368d983c68cb25f80e64e41a970cc1577f1e24c2e9b9a71b2fdf17ddfe4e36"}
Sep 08 14:15:57.858920 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:57.858883    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p" event={"ID":"5a2de755-8371-4584-917a-5478ff9b74f0","Type":"ContainerStarted","Data":"8b2e2da3f1f4178d532446b69d924122947eaf83eff8bdde6563fafa32121410"}
Sep 08 14:15:57.883785 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:57.883721    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p" podStartSLOduration=2.92499307 podStartE2EDuration="5.883701272s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.958708202s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:15:57.882096384 +0000 UTC m=+3621.333857315" watchObservedRunningTime="2026-09-08 14:15:57.883701272 +0000 UTC m=+3621.335462203"
Sep 08 14:15:58.128720 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:15:58.128470    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"
Sep 08 14:16:03.127897 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:03.127857    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"
Sep 08 14:16:03.129654 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:03.129614    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p" podUID="5a2de755-8371-4584-917a-5478ff9b74f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 08 14:16:13.128112 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:13.128064    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p" podUID="5a2de755-8371-4584-917a-5478ff9b74f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 08 14:16:23.129188 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:23.129130    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p" podUID="5a2de755-8371-4584-917a-5478ff9b74f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 08 14:16:33.128877 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:33.128825    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p" podUID="5a2de755-8371-4584-917a-5478ff9b74f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 08 14:16:34.869972 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:34.869935    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"]
Sep 08 14:16:34.873882 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:34.873856    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"
Sep 08 14:16:34.877023 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:34.876991    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 08 14:16:34.877177 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:34.877060    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 08 14:16:34.877177 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:34.877082    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 08 14:16:34.877177 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:34.877144    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 08 14:16:34.877401 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:34.877385    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-d4mnv\""
Sep 08 14:16:34.877474 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:34.877426    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 14:16:34.890654 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:34.890622    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"]
Sep 08 14:16:34.970324 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:34.970278    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86830c79-54dd-438b-b383-4b204bf9bc2c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq\" (UID: \"86830c79-54dd-438b-b383-4b204bf9bc2c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"
Sep 08 14:16:34.970506 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:34.970353    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86830c79-54dd-438b-b383-4b204bf9bc2c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq\" (UID: \"86830c79-54dd-438b-b383-4b204bf9bc2c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"
Sep 08 14:16:34.970506 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:34.970402    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86830c79-54dd-438b-b383-4b204bf9bc2c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq\" (UID: \"86830c79-54dd-438b-b383-4b204bf9bc2c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"
Sep 08 14:16:34.970506 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:34.970430    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86830c79-54dd-438b-b383-4b204bf9bc2c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq\" (UID: \"86830c79-54dd-438b-b383-4b204bf9bc2c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"
Sep 08 14:16:34.970506 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:34.970463    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86830c79-54dd-438b-b383-4b204bf9bc2c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq\" (UID: \"86830c79-54dd-438b-b383-4b204bf9bc2c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"
Sep 08 14:16:34.970506 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:34.970495    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52rwk\" (UniqueName: \"kubernetes.io/projected/86830c79-54dd-438b-b383-4b204bf9bc2c-kube-api-access-52rwk\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq\" (UID: \"86830c79-54dd-438b-b383-4b204bf9bc2c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"
Sep 08 14:16:34.970713 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:34.970553    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86830c79-54dd-438b-b383-4b204bf9bc2c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq\" (UID: \"86830c79-54dd-438b-b383-4b204bf9bc2c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"
Sep 08 14:16:34.970713 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:34.970576    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86830c79-54dd-438b-b383-4b204bf9bc2c-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq\" (UID: \"86830c79-54dd-438b-b383-4b204bf9bc2c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"
Sep 08 14:16:35.071402 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:35.071364    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86830c79-54dd-438b-b383-4b204bf9bc2c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq\" (UID: \"86830c79-54dd-438b-b383-4b204bf9bc2c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"
Sep 08 14:16:35.071622 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:35.071423    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86830c79-54dd-438b-b383-4b204bf9bc2c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq\" (UID: \"86830c79-54dd-438b-b383-4b204bf9bc2c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"
Sep 08 14:16:35.071622 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:35.071458    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86830c79-54dd-438b-b383-4b204bf9bc2c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq\" (UID: \"86830c79-54dd-438b-b383-4b204bf9bc2c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"
Sep 08 14:16:35.071622 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:35.071481    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86830c79-54dd-438b-b383-4b204bf9bc2c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq\" (UID: \"86830c79-54dd-438b-b383-4b204bf9bc2c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"
Sep 08 14:16:35.071622 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:35.071521    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52rwk\" (UniqueName: \"kubernetes.io/projected/86830c79-54dd-438b-b383-4b204bf9bc2c-kube-api-access-52rwk\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq\" (UID: \"86830c79-54dd-438b-b383-4b204bf9bc2c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"
Sep 08 14:16:35.071622 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:35.071579    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86830c79-54dd-438b-b383-4b204bf9bc2c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq\" (UID: \"86830c79-54dd-438b-b383-4b204bf9bc2c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"
Sep 08 14:16:35.071622 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:35.071615    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86830c79-54dd-438b-b383-4b204bf9bc2c-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq\" (UID: \"86830c79-54dd-438b-b383-4b204bf9bc2c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"
Sep 08 14:16:35.071953 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:35.071676    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86830c79-54dd-438b-b383-4b204bf9bc2c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq\" (UID: \"86830c79-54dd-438b-b383-4b204bf9bc2c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"
Sep 08 14:16:35.071953 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:35.071924    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86830c79-54dd-438b-b383-4b204bf9bc2c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq\" (UID: \"86830c79-54dd-438b-b383-4b204bf9bc2c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"
Sep 08 14:16:35.072231 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:35.072208    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86830c79-54dd-438b-b383-4b204bf9bc2c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq\" (UID: \"86830c79-54dd-438b-b383-4b204bf9bc2c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"
Sep 08 14:16:35.072366 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:35.072260    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86830c79-54dd-438b-b383-4b204bf9bc2c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq\" (UID: \"86830c79-54dd-438b-b383-4b204bf9bc2c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"
Sep 08 14:16:35.072436 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:35.072384    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86830c79-54dd-438b-b383-4b204bf9bc2c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq\" (UID: \"86830c79-54dd-438b-b383-4b204bf9bc2c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"
Sep 08 14:16:35.072811 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:35.072787    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86830c79-54dd-438b-b383-4b204bf9bc2c-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq\" (UID: \"86830c79-54dd-438b-b383-4b204bf9bc2c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"
Sep 08 14:16:35.074480 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:35.074443    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86830c79-54dd-438b-b383-4b204bf9bc2c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq\" (UID: \"86830c79-54dd-438b-b383-4b204bf9bc2c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"
Sep 08 14:16:35.075043 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:35.075022    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86830c79-54dd-438b-b383-4b204bf9bc2c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq\" (UID: \"86830c79-54dd-438b-b383-4b204bf9bc2c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"
Sep 08 14:16:35.090512 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:35.090479    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52rwk\" (UniqueName: \"kubernetes.io/projected/86830c79-54dd-438b-b383-4b204bf9bc2c-kube-api-access-52rwk\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq\" (UID: \"86830c79-54dd-438b-b383-4b204bf9bc2c\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"
Sep 08 14:16:35.184671 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:35.184561    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"
Sep 08 14:16:35.507649 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:35.507619    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"]
Sep 08 14:16:35.508829 ip-10-0-128-147 kubenswrapper[2558]: W0908 14:16:35.508803    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86830c79_54dd_438b_b383_4b204bf9bc2c.slice/crio-1717c7ff8a4b72bf8a99da9dbc128afb61c1826ba24e480e502e7a2c102d6670 WatchSource:0}: Error finding container 1717c7ff8a4b72bf8a99da9dbc128afb61c1826ba24e480e502e7a2c102d6670: Status 404 returned error can't find the container with id 1717c7ff8a4b72bf8a99da9dbc128afb61c1826ba24e480e502e7a2c102d6670
Sep 08 14:16:36.054363 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:36.054315    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq" event={"ID":"86830c79-54dd-438b-b383-4b204bf9bc2c","Type":"ContainerStarted","Data":"2e2272b0f3eb97054b49c7f5d2b1fdad97b19c1d1b87ab065e832dff857722b3"}
Sep 08 14:16:36.054878 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:36.054423    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq" event={"ID":"86830c79-54dd-438b-b383-4b204bf9bc2c","Type":"ContainerStarted","Data":"1717c7ff8a4b72bf8a99da9dbc128afb61c1826ba24e480e502e7a2c102d6670"}
Sep 08 14:16:39.077549 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:39.077515    2558 generic.go:356] "Generic (PLEG): container finished" podID="86830c79-54dd-438b-b383-4b204bf9bc2c" containerID="2e2272b0f3eb97054b49c7f5d2b1fdad97b19c1d1b87ab065e832dff857722b3" exitCode=0
Sep 08 14:16:39.077966 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:39.077610    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq" event={"ID":"86830c79-54dd-438b-b383-4b204bf9bc2c","Type":"ContainerDied","Data":"2e2272b0f3eb97054b49c7f5d2b1fdad97b19c1d1b87ab065e832dff857722b3"}
Sep 08 14:16:40.086381 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:40.086340    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq" event={"ID":"86830c79-54dd-438b-b383-4b204bf9bc2c","Type":"ContainerStarted","Data":"b57c0f997d3bfb5d7fdc605a5b69fb21a6048c8e5de2b3bcf1d112ed069a9cd6"}
Sep 08 14:16:40.107532 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:40.107469    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq" podStartSLOduration=3.649073605 podStartE2EDuration="6.107446582s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.458372977s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:16:40.106807859 +0000 UTC m=+3663.558568804" watchObservedRunningTime="2026-09-08 14:16:40.107446582 +0000 UTC m=+3663.559207493"
Sep 08 14:16:43.128742 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:43.128680    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p" podUID="5a2de755-8371-4584-917a-5478ff9b74f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 08 14:16:45.185121 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:45.185076    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"
Sep 08 14:16:45.185624 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:45.185157    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"
Sep 08 14:16:45.186949 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:45.186911    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq" podUID="86830c79-54dd-438b-b383-4b204bf9bc2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 08 14:16:53.129224 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:53.129173    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p" podUID="5a2de755-8371-4584-917a-5478ff9b74f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 08 14:16:55.185504 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:16:55.185461    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq" podUID="86830c79-54dd-438b-b383-4b204bf9bc2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 08 14:17:03.128841 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:17:03.128737    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p" podUID="5a2de755-8371-4584-917a-5478ff9b74f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 08 14:17:05.185451 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:17:05.185398    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq" podUID="86830c79-54dd-438b-b383-4b204bf9bc2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 08 14:17:13.129144 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:17:13.129092    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p" podUID="5a2de755-8371-4584-917a-5478ff9b74f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 08 14:17:15.185324 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:17:15.185274    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq" podUID="86830c79-54dd-438b-b383-4b204bf9bc2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 08 14:17:23.138629 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:17:23.138580    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"
Sep 08 14:17:23.146535 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:17:23.146502    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"
Sep 08 14:17:25.185137 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:17:25.185095    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq" podUID="86830c79-54dd-438b-b383-4b204bf9bc2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 08 14:17:35.185257 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:17:35.185204    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq" podUID="86830c79-54dd-438b-b383-4b204bf9bc2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 08 14:17:45.185876 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:17:45.185827    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq" podUID="86830c79-54dd-438b-b383-4b204bf9bc2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 08 14:17:49.349891 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:17:49.349841    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-57b89d5f64-vtz4p_5a2de755-8371-4584-917a-5478ff9b74f0/storage-initializer/0.log"
Sep 08 14:17:49.360300 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:17:49.360262    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-57b89d5f64-vtz4p_5a2de755-8371-4584-917a-5478ff9b74f0/main/0.log"
Sep 08 14:17:51.010382 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:17:51.010346    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"]
Sep 08 14:17:55.185522 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:17:55.185467    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq" podUID="86830c79-54dd-438b-b383-4b204bf9bc2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 08 14:18:05.185302 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:05.185254    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq" podUID="86830c79-54dd-438b-b383-4b204bf9bc2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 08 14:18:06.033643 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:06.033563    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p" podUID="5a2de755-8371-4584-917a-5478ff9b74f0" containerName="main" containerID="cri-o://8b2e2da3f1f4178d532446b69d924122947eaf83eff8bdde6563fafa32121410" gracePeriod=45
Sep 08 14:18:06.139755 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:06.139701    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p" podUID="5a2de755-8371-4584-917a-5478ff9b74f0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 08 14:18:07.091786 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.091753    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"
Sep 08 14:18:07.112681 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.112648    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a2de755-8371-4584-917a-5478ff9b74f0-kserve-provision-location\") pod \"5a2de755-8371-4584-917a-5478ff9b74f0\" (UID: \"5a2de755-8371-4584-917a-5478ff9b74f0\") "
Sep 08 14:18:07.112681 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.112687    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a2de755-8371-4584-917a-5478ff9b74f0-tls-certs\") pod \"5a2de755-8371-4584-917a-5478ff9b74f0\" (UID: \"5a2de755-8371-4584-917a-5478ff9b74f0\") "
Sep 08 14:18:07.112931 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.112704    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a2de755-8371-4584-917a-5478ff9b74f0-cabundle-cert\") pod \"5a2de755-8371-4584-917a-5478ff9b74f0\" (UID: \"5a2de755-8371-4584-917a-5478ff9b74f0\") "
Sep 08 14:18:07.112931 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.112723    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a2de755-8371-4584-917a-5478ff9b74f0-home\") pod \"5a2de755-8371-4584-917a-5478ff9b74f0\" (UID: \"5a2de755-8371-4584-917a-5478ff9b74f0\") "
Sep 08 14:18:07.112931 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.112795    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dzph\" (UniqueName: \"kubernetes.io/projected/5a2de755-8371-4584-917a-5478ff9b74f0-kube-api-access-2dzph\") pod \"5a2de755-8371-4584-917a-5478ff9b74f0\" (UID: \"5a2de755-8371-4584-917a-5478ff9b74f0\") "
Sep 08 14:18:07.112931 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.112823    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a2de755-8371-4584-917a-5478ff9b74f0-model-cache\") pod \"5a2de755-8371-4584-917a-5478ff9b74f0\" (UID: \"5a2de755-8371-4584-917a-5478ff9b74f0\") "
Sep 08 14:18:07.112931 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.112851    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a2de755-8371-4584-917a-5478ff9b74f0-tmp-dir\") pod \"5a2de755-8371-4584-917a-5478ff9b74f0\" (UID: \"5a2de755-8371-4584-917a-5478ff9b74f0\") "
Sep 08 14:18:07.112931 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.112930    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a2de755-8371-4584-917a-5478ff9b74f0-dshm\") pod \"5a2de755-8371-4584-917a-5478ff9b74f0\" (UID: \"5a2de755-8371-4584-917a-5478ff9b74f0\") "
Sep 08 14:18:07.113269 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.113081    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a2de755-8371-4584-917a-5478ff9b74f0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5a2de755-8371-4584-917a-5478ff9b74f0" (UID: "5a2de755-8371-4584-917a-5478ff9b74f0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:18:07.113269 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.113137    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a2de755-8371-4584-917a-5478ff9b74f0-home" (OuterVolumeSpecName: "home") pod "5a2de755-8371-4584-917a-5478ff9b74f0" (UID: "5a2de755-8371-4584-917a-5478ff9b74f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:18:07.113269 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.113243    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a2de755-8371-4584-917a-5478ff9b74f0-cabundle-cert\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:18:07.113269 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.113263    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a2de755-8371-4584-917a-5478ff9b74f0-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:18:07.113489 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.113448    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a2de755-8371-4584-917a-5478ff9b74f0-model-cache" (OuterVolumeSpecName: "model-cache") pod "5a2de755-8371-4584-917a-5478ff9b74f0" (UID: "5a2de755-8371-4584-917a-5478ff9b74f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:18:07.116069 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.115669    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a2de755-8371-4584-917a-5478ff9b74f0-dshm" (OuterVolumeSpecName: "dshm") pod "5a2de755-8371-4584-917a-5478ff9b74f0" (UID: "5a2de755-8371-4584-917a-5478ff9b74f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:18:07.116069 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.115970    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a2de755-8371-4584-917a-5478ff9b74f0-kube-api-access-2dzph" (OuterVolumeSpecName: "kube-api-access-2dzph") pod "5a2de755-8371-4584-917a-5478ff9b74f0" (UID: "5a2de755-8371-4584-917a-5478ff9b74f0"). InnerVolumeSpecName "kube-api-access-2dzph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:18:07.118169 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.118119    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a2de755-8371-4584-917a-5478ff9b74f0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5a2de755-8371-4584-917a-5478ff9b74f0" (UID: "5a2de755-8371-4584-917a-5478ff9b74f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:18:07.131243 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.131181    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a2de755-8371-4584-917a-5478ff9b74f0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5a2de755-8371-4584-917a-5478ff9b74f0" (UID: "5a2de755-8371-4584-917a-5478ff9b74f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:18:07.142330 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.142296    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a2de755-8371-4584-917a-5478ff9b74f0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5a2de755-8371-4584-917a-5478ff9b74f0" (UID: "5a2de755-8371-4584-917a-5478ff9b74f0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:18:07.214604 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.214566    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a2de755-8371-4584-917a-5478ff9b74f0-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:18:07.214764 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.214612    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a2de755-8371-4584-917a-5478ff9b74f0-kserve-provision-location\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:18:07.214764 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.214623    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a2de755-8371-4584-917a-5478ff9b74f0-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:18:07.214764 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.214634    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dzph\" (UniqueName: \"kubernetes.io/projected/5a2de755-8371-4584-917a-5478ff9b74f0-kube-api-access-2dzph\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:18:07.214764 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.214643    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a2de755-8371-4584-917a-5478ff9b74f0-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:18:07.214764 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.214650    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a2de755-8371-4584-917a-5478ff9b74f0-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:18:07.573812 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.573726    2558 generic.go:356] "Generic (PLEG): container finished" podID="5a2de755-8371-4584-917a-5478ff9b74f0" containerID="8b2e2da3f1f4178d532446b69d924122947eaf83eff8bdde6563fafa32121410" exitCode=0
Sep 08 14:18:07.573812 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.573779    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"
Sep 08 14:18:07.573996 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.573809    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p" event={"ID":"5a2de755-8371-4584-917a-5478ff9b74f0","Type":"ContainerDied","Data":"8b2e2da3f1f4178d532446b69d924122947eaf83eff8bdde6563fafa32121410"}
Sep 08 14:18:07.573996 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.573866    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p" event={"ID":"5a2de755-8371-4584-917a-5478ff9b74f0","Type":"ContainerDied","Data":"3a3a2eef82f5121e06abc871b5bc375f7239b0e8d7a6da5cc0535d55e1afe23d"}
Sep 08 14:18:07.573996 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.573896    2558 scope.go:117] "RemoveContainer" containerID="8b2e2da3f1f4178d532446b69d924122947eaf83eff8bdde6563fafa32121410"
Sep 08 14:18:07.585295 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.585273    2558 scope.go:117] "RemoveContainer" containerID="b2368d983c68cb25f80e64e41a970cc1577f1e24c2e9b9a71b2fdf17ddfe4e36"
Sep 08 14:18:07.593830 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.593780    2558 scope.go:117] "RemoveContainer" containerID="8b2e2da3f1f4178d532446b69d924122947eaf83eff8bdde6563fafa32121410"
Sep 08 14:18:07.594157 ip-10-0-128-147 kubenswrapper[2558]: E0908 14:18:07.594130    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b2e2da3f1f4178d532446b69d924122947eaf83eff8bdde6563fafa32121410\": container with ID starting with 8b2e2da3f1f4178d532446b69d924122947eaf83eff8bdde6563fafa32121410 not found: ID does not exist" containerID="8b2e2da3f1f4178d532446b69d924122947eaf83eff8bdde6563fafa32121410"
Sep 08 14:18:07.594277 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.594168    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b2e2da3f1f4178d532446b69d924122947eaf83eff8bdde6563fafa32121410"} err="failed to get container status \"8b2e2da3f1f4178d532446b69d924122947eaf83eff8bdde6563fafa32121410\": rpc error: code = NotFound desc = could not find container \"8b2e2da3f1f4178d532446b69d924122947eaf83eff8bdde6563fafa32121410\": container with ID starting with 8b2e2da3f1f4178d532446b69d924122947eaf83eff8bdde6563fafa32121410 not found: ID does not exist"
Sep 08 14:18:07.594277 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.594192    2558 scope.go:117] "RemoveContainer" containerID="b2368d983c68cb25f80e64e41a970cc1577f1e24c2e9b9a71b2fdf17ddfe4e36"
Sep 08 14:18:07.594549 ip-10-0-128-147 kubenswrapper[2558]: E0908 14:18:07.594521    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2368d983c68cb25f80e64e41a970cc1577f1e24c2e9b9a71b2fdf17ddfe4e36\": container with ID starting with b2368d983c68cb25f80e64e41a970cc1577f1e24c2e9b9a71b2fdf17ddfe4e36 not found: ID does not exist" containerID="b2368d983c68cb25f80e64e41a970cc1577f1e24c2e9b9a71b2fdf17ddfe4e36"
Sep 08 14:18:07.594657 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.594559    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2368d983c68cb25f80e64e41a970cc1577f1e24c2e9b9a71b2fdf17ddfe4e36"} err="failed to get container status \"b2368d983c68cb25f80e64e41a970cc1577f1e24c2e9b9a71b2fdf17ddfe4e36\": rpc error: code = NotFound desc = could not find container \"b2368d983c68cb25f80e64e41a970cc1577f1e24c2e9b9a71b2fdf17ddfe4e36\": container with ID starting with b2368d983c68cb25f80e64e41a970cc1577f1e24c2e9b9a71b2fdf17ddfe4e36 not found: ID does not exist"
Sep 08 14:18:07.596503 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.596481    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"]
Sep 08 14:18:07.603351 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:07.603322    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-57b89d5f64-vtz4p"]
Sep 08 14:18:09.054870 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:09.054834    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a2de755-8371-4584-917a-5478ff9b74f0" path="/var/lib/kubelet/pods/5a2de755-8371-4584-917a-5478ff9b74f0/volumes"
Sep 08 14:18:15.185049 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:15.184999    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq" podUID="86830c79-54dd-438b-b383-4b204bf9bc2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 08 14:18:16.977849 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:16.977814    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"]
Sep 08 14:18:16.978219 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:16.978182    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a2de755-8371-4584-917a-5478ff9b74f0" containerName="main"
Sep 08 14:18:16.978219 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:16.978196    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a2de755-8371-4584-917a-5478ff9b74f0" containerName="main"
Sep 08 14:18:16.978219 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:16.978217    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a2de755-8371-4584-917a-5478ff9b74f0" containerName="storage-initializer"
Sep 08 14:18:16.978351 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:16.978223    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a2de755-8371-4584-917a-5478ff9b74f0" containerName="storage-initializer"
Sep 08 14:18:16.978351 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:16.978284    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a2de755-8371-4584-917a-5478ff9b74f0" containerName="main"
Sep 08 14:18:16.983097 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:16.983080    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"
Sep 08 14:18:16.985752 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:16.985721    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 14:18:16.985952 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:16.985723    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Sep 08 14:18:16.986096 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:16.985721    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Sep 08 14:18:16.986096 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:16.985721    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-k9v9m\""
Sep 08 14:18:16.986323 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:16.986303    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Sep 08 14:18:16.986449 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:16.986434    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Sep 08 14:18:16.998210 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:16.998186    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"]
Sep 08 14:18:17.105817 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:17.105772    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-95967666f-qtbxs\" (UID: \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"
Sep 08 14:18:17.106006 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:17.105895    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfjnv\" (UniqueName: \"kubernetes.io/projected/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-kube-api-access-qfjnv\") pod \"lora-multiple-adapters-test-kserve-95967666f-qtbxs\" (UID: \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"
Sep 08 14:18:17.106006 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:17.105962    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-95967666f-qtbxs\" (UID: \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"
Sep 08 14:18:17.106129 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:17.106011    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-tls-certs\") pod \"lora-multiple-adapters-test-kserve-95967666f-qtbxs\" (UID: \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"
Sep 08 14:18:17.106129 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:17.106034    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-95967666f-qtbxs\" (UID: \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"
Sep 08 14:18:17.106129 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:17.106061    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-model-cache\") pod \"lora-multiple-adapters-test-kserve-95967666f-qtbxs\" (UID: \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"
Sep 08 14:18:17.106129 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:17.106082    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-home\") pod \"lora-multiple-adapters-test-kserve-95967666f-qtbxs\" (UID: \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"
Sep 08 14:18:17.106359 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:17.106150    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-dshm\") pod \"lora-multiple-adapters-test-kserve-95967666f-qtbxs\" (UID: \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"
Sep 08 14:18:17.206896 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:17.206857    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-tls-certs\") pod \"lora-multiple-adapters-test-kserve-95967666f-qtbxs\" (UID: \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"
Sep 08 14:18:17.206896 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:17.206897    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-95967666f-qtbxs\" (UID: \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"
Sep 08 14:18:17.207152 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:17.206920    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-model-cache\") pod \"lora-multiple-adapters-test-kserve-95967666f-qtbxs\" (UID: \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"
Sep 08 14:18:17.207152 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:17.206940    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-home\") pod \"lora-multiple-adapters-test-kserve-95967666f-qtbxs\" (UID: \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"
Sep 08 14:18:17.207152 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:17.206983    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-dshm\") pod \"lora-multiple-adapters-test-kserve-95967666f-qtbxs\" (UID: \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"
Sep 08 14:18:17.207152 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:17.207077    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-95967666f-qtbxs\" (UID: \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"
Sep 08 14:18:17.207152 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:17.207116    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfjnv\" (UniqueName: \"kubernetes.io/projected/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-kube-api-access-qfjnv\") pod \"lora-multiple-adapters-test-kserve-95967666f-qtbxs\" (UID: \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"
Sep 08 14:18:17.207425 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:17.207182    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-95967666f-qtbxs\" (UID: \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"
Sep 08 14:18:17.207425 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:17.207372    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-home\") pod \"lora-multiple-adapters-test-kserve-95967666f-qtbxs\" (UID: \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"
Sep 08 14:18:17.207543 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:17.207455    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-95967666f-qtbxs\" (UID: \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"
Sep 08 14:18:17.207543 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:17.207499    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-95967666f-qtbxs\" (UID: \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"
Sep 08 14:18:17.207673 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:17.207639    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-model-cache\") pod \"lora-multiple-adapters-test-kserve-95967666f-qtbxs\" (UID: \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"
Sep 08 14:18:17.207831 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:17.207801    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-95967666f-qtbxs\" (UID: \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"
Sep 08 14:18:17.209606 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:17.209567    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-dshm\") pod \"lora-multiple-adapters-test-kserve-95967666f-qtbxs\" (UID: \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"
Sep 08 14:18:17.209731 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:17.209647    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-tls-certs\") pod \"lora-multiple-adapters-test-kserve-95967666f-qtbxs\" (UID: \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"
Sep 08 14:18:17.220663 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:17.220613    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfjnv\" (UniqueName: \"kubernetes.io/projected/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-kube-api-access-qfjnv\") pod \"lora-multiple-adapters-test-kserve-95967666f-qtbxs\" (UID: \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"
Sep 08 14:18:17.293758 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:17.293666    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"
Sep 08 14:18:17.585058 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:17.585023    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"]
Sep 08 14:18:17.587257 ip-10-0-128-147 kubenswrapper[2558]: W0908 14:18:17.587220    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e599e63_cbb3_4528_8d0f_d451eacfb3e6.slice/crio-fcdd9f92d2d8cfa1d830922bbba684e2da81c7aec720acc50fb27220b5f3e3ea WatchSource:0}: Error finding container fcdd9f92d2d8cfa1d830922bbba684e2da81c7aec720acc50fb27220b5f3e3ea: Status 404 returned error can't find the container with id fcdd9f92d2d8cfa1d830922bbba684e2da81c7aec720acc50fb27220b5f3e3ea
Sep 08 14:18:17.626804 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:17.626761    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs" event={"ID":"4e599e63-cbb3-4528-8d0f-d451eacfb3e6","Type":"ContainerStarted","Data":"fcdd9f92d2d8cfa1d830922bbba684e2da81c7aec720acc50fb27220b5f3e3ea"}
Sep 08 14:18:18.634153 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:18.634109    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs" event={"ID":"4e599e63-cbb3-4528-8d0f-d451eacfb3e6","Type":"ContainerStarted","Data":"f8cae5a0ae45081d77f8a9a2c39e861acca40c80e8e611982017260b15823428"}
Sep 08 14:18:21.650728 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:21.650689    2558 generic.go:356] "Generic (PLEG): container finished" podID="4e599e63-cbb3-4528-8d0f-d451eacfb3e6" containerID="f8cae5a0ae45081d77f8a9a2c39e861acca40c80e8e611982017260b15823428" exitCode=0
Sep 08 14:18:21.651174 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:21.650737    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs" event={"ID":"4e599e63-cbb3-4528-8d0f-d451eacfb3e6","Type":"ContainerDied","Data":"f8cae5a0ae45081d77f8a9a2c39e861acca40c80e8e611982017260b15823428"}
Sep 08 14:18:22.657387 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:22.657347    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs" event={"ID":"4e599e63-cbb3-4528-8d0f-d451eacfb3e6","Type":"ContainerStarted","Data":"8844f90ab250f8c94274f16e8792616d95c9227c5ea731aeafeaa464858a044b"}
Sep 08 14:18:22.682067 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:22.682003    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs" podStartSLOduration=2.880812944 podStartE2EDuration="6.681981239s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.801168295s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:18:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:18:22.679814214 +0000 UTC m=+3766.131575145" watchObservedRunningTime="2026-09-08 14:18:22.681981239 +0000 UTC m=+3766.133742171"
Sep 08 14:18:23.294485 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:23.294447    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"
Sep 08 14:18:25.185471 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:25.185425    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq" podUID="86830c79-54dd-438b-b383-4b204bf9bc2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 08 14:18:27.294850 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:27.294809    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"
Sep 08 14:18:27.296585 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:27.296547    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs" podUID="4e599e63-cbb3-4528-8d0f-d451eacfb3e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 08 14:18:35.185401 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:35.185291    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq" podUID="86830c79-54dd-438b-b383-4b204bf9bc2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Sep 08 14:18:37.294496 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:37.294443    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs" podUID="4e599e63-cbb3-4528-8d0f-d451eacfb3e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 08 14:18:45.195213 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:45.195169    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"
Sep 08 14:18:45.203162 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:45.203129    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"
Sep 08 14:18:47.294308 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:47.294257    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs" podUID="4e599e63-cbb3-4528-8d0f-d451eacfb3e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 08 14:18:54.363807 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:54.363761    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"]
Sep 08 14:18:57.294637 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:18:57.294556    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs" podUID="4e599e63-cbb3-4528-8d0f-d451eacfb3e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 08 14:19:07.294638 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:07.294561    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs" podUID="4e599e63-cbb3-4528-8d0f-d451eacfb3e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 08 14:19:09.395164 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:09.395115    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq" podUID="86830c79-54dd-438b-b383-4b204bf9bc2c" containerName="main" containerID="cri-o://b57c0f997d3bfb5d7fdc605a5b69fb21a6048c8e5de2b3bcf1d112ed069a9cd6" gracePeriod=45
Sep 08 14:19:10.148467 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.148441    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"
Sep 08 14:19:10.286321 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.286223    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86830c79-54dd-438b-b383-4b204bf9bc2c-home\") pod \"86830c79-54dd-438b-b383-4b204bf9bc2c\" (UID: \"86830c79-54dd-438b-b383-4b204bf9bc2c\") "
Sep 08 14:19:10.286321 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.286277    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86830c79-54dd-438b-b383-4b204bf9bc2c-dshm\") pod \"86830c79-54dd-438b-b383-4b204bf9bc2c\" (UID: \"86830c79-54dd-438b-b383-4b204bf9bc2c\") "
Sep 08 14:19:10.286571 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.286322    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86830c79-54dd-438b-b383-4b204bf9bc2c-cabundle-cert\") pod \"86830c79-54dd-438b-b383-4b204bf9bc2c\" (UID: \"86830c79-54dd-438b-b383-4b204bf9bc2c\") "
Sep 08 14:19:10.286571 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.286359    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86830c79-54dd-438b-b383-4b204bf9bc2c-tls-certs\") pod \"86830c79-54dd-438b-b383-4b204bf9bc2c\" (UID: \"86830c79-54dd-438b-b383-4b204bf9bc2c\") "
Sep 08 14:19:10.286571 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.286380    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86830c79-54dd-438b-b383-4b204bf9bc2c-tmp-dir\") pod \"86830c79-54dd-438b-b383-4b204bf9bc2c\" (UID: \"86830c79-54dd-438b-b383-4b204bf9bc2c\") "
Sep 08 14:19:10.286571 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.286420    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86830c79-54dd-438b-b383-4b204bf9bc2c-kserve-provision-location\") pod \"86830c79-54dd-438b-b383-4b204bf9bc2c\" (UID: \"86830c79-54dd-438b-b383-4b204bf9bc2c\") "
Sep 08 14:19:10.286571 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.286455    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86830c79-54dd-438b-b383-4b204bf9bc2c-model-cache\") pod \"86830c79-54dd-438b-b383-4b204bf9bc2c\" (UID: \"86830c79-54dd-438b-b383-4b204bf9bc2c\") "
Sep 08 14:19:10.286571 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.286505    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52rwk\" (UniqueName: \"kubernetes.io/projected/86830c79-54dd-438b-b383-4b204bf9bc2c-kube-api-access-52rwk\") pod \"86830c79-54dd-438b-b383-4b204bf9bc2c\" (UID: \"86830c79-54dd-438b-b383-4b204bf9bc2c\") "
Sep 08 14:19:10.286916 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.286796    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86830c79-54dd-438b-b383-4b204bf9bc2c-model-cache" (OuterVolumeSpecName: "model-cache") pod "86830c79-54dd-438b-b383-4b204bf9bc2c" (UID: "86830c79-54dd-438b-b383-4b204bf9bc2c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:19:10.287045 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.287015    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86830c79-54dd-438b-b383-4b204bf9bc2c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "86830c79-54dd-438b-b383-4b204bf9bc2c" (UID: "86830c79-54dd-438b-b383-4b204bf9bc2c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:19:10.287128 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.287027    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86830c79-54dd-438b-b383-4b204bf9bc2c-home" (OuterVolumeSpecName: "home") pod "86830c79-54dd-438b-b383-4b204bf9bc2c" (UID: "86830c79-54dd-438b-b383-4b204bf9bc2c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:19:10.289321 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.289286    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86830c79-54dd-438b-b383-4b204bf9bc2c-dshm" (OuterVolumeSpecName: "dshm") pod "86830c79-54dd-438b-b383-4b204bf9bc2c" (UID: "86830c79-54dd-438b-b383-4b204bf9bc2c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:19:10.289321 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.289303    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86830c79-54dd-438b-b383-4b204bf9bc2c-kube-api-access-52rwk" (OuterVolumeSpecName: "kube-api-access-52rwk") pod "86830c79-54dd-438b-b383-4b204bf9bc2c" (UID: "86830c79-54dd-438b-b383-4b204bf9bc2c"). InnerVolumeSpecName "kube-api-access-52rwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:19:10.289512 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.289358    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86830c79-54dd-438b-b383-4b204bf9bc2c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "86830c79-54dd-438b-b383-4b204bf9bc2c" (UID: "86830c79-54dd-438b-b383-4b204bf9bc2c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:19:10.300239 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.300202    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86830c79-54dd-438b-b383-4b204bf9bc2c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "86830c79-54dd-438b-b383-4b204bf9bc2c" (UID: "86830c79-54dd-438b-b383-4b204bf9bc2c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:19:10.309232 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.309199    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86830c79-54dd-438b-b383-4b204bf9bc2c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "86830c79-54dd-438b-b383-4b204bf9bc2c" (UID: "86830c79-54dd-438b-b383-4b204bf9bc2c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:19:10.387749 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.387716    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86830c79-54dd-438b-b383-4b204bf9bc2c-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:19:10.387749 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.387746    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86830c79-54dd-438b-b383-4b204bf9bc2c-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:19:10.387749 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.387756    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86830c79-54dd-438b-b383-4b204bf9bc2c-kserve-provision-location\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:19:10.387965 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.387766    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86830c79-54dd-438b-b383-4b204bf9bc2c-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:19:10.387965 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.387775    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52rwk\" (UniqueName: \"kubernetes.io/projected/86830c79-54dd-438b-b383-4b204bf9bc2c-kube-api-access-52rwk\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:19:10.387965 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.387794    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86830c79-54dd-438b-b383-4b204bf9bc2c-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:19:10.387965 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.387806    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86830c79-54dd-438b-b383-4b204bf9bc2c-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:19:10.387965 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.387814    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86830c79-54dd-438b-b383-4b204bf9bc2c-cabundle-cert\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:19:10.922633 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.922581    2558 generic.go:356] "Generic (PLEG): container finished" podID="86830c79-54dd-438b-b383-4b204bf9bc2c" containerID="b57c0f997d3bfb5d7fdc605a5b69fb21a6048c8e5de2b3bcf1d112ed069a9cd6" exitCode=0
Sep 08 14:19:10.923057 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.922677    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"
Sep 08 14:19:10.923057 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.922689    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq" event={"ID":"86830c79-54dd-438b-b383-4b204bf9bc2c","Type":"ContainerDied","Data":"b57c0f997d3bfb5d7fdc605a5b69fb21a6048c8e5de2b3bcf1d112ed069a9cd6"}
Sep 08 14:19:10.923057 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.922782    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq" event={"ID":"86830c79-54dd-438b-b383-4b204bf9bc2c","Type":"ContainerDied","Data":"1717c7ff8a4b72bf8a99da9dbc128afb61c1826ba24e480e502e7a2c102d6670"}
Sep 08 14:19:10.923057 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.922813    2558 scope.go:117] "RemoveContainer" containerID="b57c0f997d3bfb5d7fdc605a5b69fb21a6048c8e5de2b3bcf1d112ed069a9cd6"
Sep 08 14:19:10.932766 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.932746    2558 scope.go:117] "RemoveContainer" containerID="2e2272b0f3eb97054b49c7f5d2b1fdad97b19c1d1b87ab065e832dff857722b3"
Sep 08 14:19:10.943390 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.943362    2558 scope.go:117] "RemoveContainer" containerID="b57c0f997d3bfb5d7fdc605a5b69fb21a6048c8e5de2b3bcf1d112ed069a9cd6"
Sep 08 14:19:10.943794 ip-10-0-128-147 kubenswrapper[2558]: E0908 14:19:10.943765    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b57c0f997d3bfb5d7fdc605a5b69fb21a6048c8e5de2b3bcf1d112ed069a9cd6\": container with ID starting with b57c0f997d3bfb5d7fdc605a5b69fb21a6048c8e5de2b3bcf1d112ed069a9cd6 not found: ID does not exist" containerID="b57c0f997d3bfb5d7fdc605a5b69fb21a6048c8e5de2b3bcf1d112ed069a9cd6"
Sep 08 14:19:10.943883 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.943801    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b57c0f997d3bfb5d7fdc605a5b69fb21a6048c8e5de2b3bcf1d112ed069a9cd6"} err="failed to get container status \"b57c0f997d3bfb5d7fdc605a5b69fb21a6048c8e5de2b3bcf1d112ed069a9cd6\": rpc error: code = NotFound desc = could not find container \"b57c0f997d3bfb5d7fdc605a5b69fb21a6048c8e5de2b3bcf1d112ed069a9cd6\": container with ID starting with b57c0f997d3bfb5d7fdc605a5b69fb21a6048c8e5de2b3bcf1d112ed069a9cd6 not found: ID does not exist"
Sep 08 14:19:10.943883 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.943822    2558 scope.go:117] "RemoveContainer" containerID="2e2272b0f3eb97054b49c7f5d2b1fdad97b19c1d1b87ab065e832dff857722b3"
Sep 08 14:19:10.944207 ip-10-0-128-147 kubenswrapper[2558]: E0908 14:19:10.944181    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e2272b0f3eb97054b49c7f5d2b1fdad97b19c1d1b87ab065e832dff857722b3\": container with ID starting with 2e2272b0f3eb97054b49c7f5d2b1fdad97b19c1d1b87ab065e832dff857722b3 not found: ID does not exist" containerID="2e2272b0f3eb97054b49c7f5d2b1fdad97b19c1d1b87ab065e832dff857722b3"
Sep 08 14:19:10.944312 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.944217    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e2272b0f3eb97054b49c7f5d2b1fdad97b19c1d1b87ab065e832dff857722b3"} err="failed to get container status \"2e2272b0f3eb97054b49c7f5d2b1fdad97b19c1d1b87ab065e832dff857722b3\": rpc error: code = NotFound desc = could not find container \"2e2272b0f3eb97054b49c7f5d2b1fdad97b19c1d1b87ab065e832dff857722b3\": container with ID starting with 2e2272b0f3eb97054b49c7f5d2b1fdad97b19c1d1b87ab065e832dff857722b3 not found: ID does not exist"
Sep 08 14:19:10.946172 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.946151    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"]
Sep 08 14:19:10.949646 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:10.949621    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jp2kq"]
Sep 08 14:19:11.053871 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:11.053830    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86830c79-54dd-438b-b383-4b204bf9bc2c" path="/var/lib/kubelet/pods/86830c79-54dd-438b-b383-4b204bf9bc2c/volumes"
Sep 08 14:19:17.294515 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:17.294464    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs" podUID="4e599e63-cbb3-4528-8d0f-d451eacfb3e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 08 14:19:27.294649 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:27.294567    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs" podUID="4e599e63-cbb3-4528-8d0f-d451eacfb3e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 08 14:19:37.294578 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:37.294521    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs" podUID="4e599e63-cbb3-4528-8d0f-d451eacfb3e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 08 14:19:47.304579 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:47.304543    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"
Sep 08 14:19:47.312667 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:19:47.312626    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"
Sep 08 14:20:12.310866 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.310791    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"]
Sep 08 14:20:12.311254 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.311159    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86830c79-54dd-438b-b383-4b204bf9bc2c" containerName="main"
Sep 08 14:20:12.311254 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.311171    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="86830c79-54dd-438b-b383-4b204bf9bc2c" containerName="main"
Sep 08 14:20:12.311254 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.311190    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86830c79-54dd-438b-b383-4b204bf9bc2c" containerName="storage-initializer"
Sep 08 14:20:12.311254 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.311196    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="86830c79-54dd-438b-b383-4b204bf9bc2c" containerName="storage-initializer"
Sep 08 14:20:12.311254 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.311247    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="86830c79-54dd-438b-b383-4b204bf9bc2c" containerName="main"
Sep 08 14:20:12.314576 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.314557    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"
Sep 08 14:20:12.318408 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.318380    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 08 14:20:12.318538 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.318418    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 14:20:12.318538 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.318479    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-rjpx4\""
Sep 08 14:20:12.318753 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.318736    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 08 14:20:12.318815 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.318799    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 08 14:20:12.318856 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.318844    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 08 14:20:12.339190 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.339154    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"]
Sep 08 14:20:12.447401 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.447360    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ff739e2-1ad2-4031-a964-1f55201f5fd5-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2\" (UID: \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"
Sep 08 14:20:12.447401 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.447404    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bzxw\" (UniqueName: \"kubernetes.io/projected/1ff739e2-1ad2-4031-a964-1f55201f5fd5-kube-api-access-5bzxw\") pod \"custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2\" (UID: \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"
Sep 08 14:20:12.447735 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.447527    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ff739e2-1ad2-4031-a964-1f55201f5fd5-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2\" (UID: \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"
Sep 08 14:20:12.447735 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.447616    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ff739e2-1ad2-4031-a964-1f55201f5fd5-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2\" (UID: \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"
Sep 08 14:20:12.447735 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.447655    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ff739e2-1ad2-4031-a964-1f55201f5fd5-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2\" (UID: \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"
Sep 08 14:20:12.447735 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.447724    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ff739e2-1ad2-4031-a964-1f55201f5fd5-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2\" (UID: \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"
Sep 08 14:20:12.447921 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.447760    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ff739e2-1ad2-4031-a964-1f55201f5fd5-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2\" (UID: \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"
Sep 08 14:20:12.447921 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.447799    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ff739e2-1ad2-4031-a964-1f55201f5fd5-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2\" (UID: \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"
Sep 08 14:20:12.548497 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.548442    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ff739e2-1ad2-4031-a964-1f55201f5fd5-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2\" (UID: \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"
Sep 08 14:20:12.548497 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.548503    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ff739e2-1ad2-4031-a964-1f55201f5fd5-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2\" (UID: \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"
Sep 08 14:20:12.548807 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.548539    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bzxw\" (UniqueName: \"kubernetes.io/projected/1ff739e2-1ad2-4031-a964-1f55201f5fd5-kube-api-access-5bzxw\") pod \"custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2\" (UID: \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"
Sep 08 14:20:12.548807 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.548732    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ff739e2-1ad2-4031-a964-1f55201f5fd5-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2\" (UID: \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"
Sep 08 14:20:12.548807 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.548800    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ff739e2-1ad2-4031-a964-1f55201f5fd5-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2\" (UID: \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"
Sep 08 14:20:12.548966 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.548844    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ff739e2-1ad2-4031-a964-1f55201f5fd5-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2\" (UID: \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"
Sep 08 14:20:12.548966 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.548889    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ff739e2-1ad2-4031-a964-1f55201f5fd5-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2\" (UID: \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"
Sep 08 14:20:12.548966 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.548929    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ff739e2-1ad2-4031-a964-1f55201f5fd5-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2\" (UID: \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"
Sep 08 14:20:12.548966 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.548935    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ff739e2-1ad2-4031-a964-1f55201f5fd5-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2\" (UID: \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"
Sep 08 14:20:12.549144 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.549040    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ff739e2-1ad2-4031-a964-1f55201f5fd5-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2\" (UID: \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"
Sep 08 14:20:12.549278 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.549262    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ff739e2-1ad2-4031-a964-1f55201f5fd5-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2\" (UID: \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"
Sep 08 14:20:12.549332 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.549296    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ff739e2-1ad2-4031-a964-1f55201f5fd5-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2\" (UID: \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"
Sep 08 14:20:12.549610 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.549564    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ff739e2-1ad2-4031-a964-1f55201f5fd5-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2\" (UID: \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"
Sep 08 14:20:12.551048 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.551029    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ff739e2-1ad2-4031-a964-1f55201f5fd5-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2\" (UID: \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"
Sep 08 14:20:12.551335 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.551317    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ff739e2-1ad2-4031-a964-1f55201f5fd5-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2\" (UID: \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"
Sep 08 14:20:12.564055 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.563986    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bzxw\" (UniqueName: \"kubernetes.io/projected/1ff739e2-1ad2-4031-a964-1f55201f5fd5-kube-api-access-5bzxw\") pod \"custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2\" (UID: \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"
Sep 08 14:20:12.625069 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.625026    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"
Sep 08 14:20:12.910249 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:12.910223    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"]
Sep 08 14:20:12.911988 ip-10-0-128-147 kubenswrapper[2558]: W0908 14:20:12.911956    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ff739e2_1ad2_4031_a964_1f55201f5fd5.slice/crio-2445d6e5a291552ad0c60c812b7ea736d585290ae122a0ce21e6f974ac779f0d WatchSource:0}: Error finding container 2445d6e5a291552ad0c60c812b7ea736d585290ae122a0ce21e6f974ac779f0d: Status 404 returned error can't find the container with id 2445d6e5a291552ad0c60c812b7ea736d585290ae122a0ce21e6f974ac779f0d
Sep 08 14:20:13.240874 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:13.240772    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2" event={"ID":"1ff739e2-1ad2-4031-a964-1f55201f5fd5","Type":"ContainerStarted","Data":"dbd7b79b254f7e13860bfd2ea26050a3e41a92cde0ff8f620c94a62ee995496d"}
Sep 08 14:20:13.240874 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:13.240841    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2" event={"ID":"1ff739e2-1ad2-4031-a964-1f55201f5fd5","Type":"ContainerStarted","Data":"2445d6e5a291552ad0c60c812b7ea736d585290ae122a0ce21e6f974ac779f0d"}
Sep 08 14:20:15.259280 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:15.259222    2558 generic.go:356] "Generic (PLEG): container finished" podID="1ff739e2-1ad2-4031-a964-1f55201f5fd5" containerID="dbd7b79b254f7e13860bfd2ea26050a3e41a92cde0ff8f620c94a62ee995496d" exitCode=0
Sep 08 14:20:15.259859 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:15.259367    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2" event={"ID":"1ff739e2-1ad2-4031-a964-1f55201f5fd5","Type":"ContainerDied","Data":"dbd7b79b254f7e13860bfd2ea26050a3e41a92cde0ff8f620c94a62ee995496d"}
Sep 08 14:20:16.266709 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:16.266675    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2" event={"ID":"1ff739e2-1ad2-4031-a964-1f55201f5fd5","Type":"ContainerStarted","Data":"f133cf7f37b4921f4ad72dbb954041af4259c64c05424794406048aefed6be98"}
Sep 08 14:20:16.313496 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:16.313435    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2" podStartSLOduration=2.077296368 podStartE2EDuration="4.313417144s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.236120776s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:20:16.309507513 +0000 UTC m=+3879.761268444" watchObservedRunningTime="2026-09-08 14:20:16.313417144 +0000 UTC m=+3879.765178117"
Sep 08 14:20:22.625209 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:22.625168    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"
Sep 08 14:20:22.625842 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:22.625253    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"
Sep 08 14:20:22.626995 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:22.626961    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2" podUID="1ff739e2-1ad2-4031-a964-1f55201f5fd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 08 14:20:32.626025 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:32.625974    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2" podUID="1ff739e2-1ad2-4031-a964-1f55201f5fd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 08 14:20:42.625938 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:42.625892    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2" podUID="1ff739e2-1ad2-4031-a964-1f55201f5fd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 08 14:20:44.894653 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:44.894617    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-95967666f-qtbxs_4e599e63-cbb3-4528-8d0f-d451eacfb3e6/storage-initializer/0.log"
Sep 08 14:20:44.904117 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:44.904091    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-95967666f-qtbxs_4e599e63-cbb3-4528-8d0f-d451eacfb3e6/main/0.log"
Sep 08 14:20:46.183263 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:46.183223    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"]
Sep 08 14:20:52.626035 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:20:52.625978    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2" podUID="1ff739e2-1ad2-4031-a964-1f55201f5fd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 08 14:21:01.206737 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:01.206696    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs" podUID="4e599e63-cbb3-4528-8d0f-d451eacfb3e6" containerName="main" containerID="cri-o://8844f90ab250f8c94274f16e8792616d95c9227c5ea731aeafeaa464858a044b" gracePeriod=45
Sep 08 14:21:01.305767 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:01.305713    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs" podUID="4e599e63-cbb3-4528-8d0f-d451eacfb3e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Sep 08 14:21:01.862066 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:01.862035    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"
Sep 08 14:21:02.015522 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.015439    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfjnv\" (UniqueName: \"kubernetes.io/projected/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-kube-api-access-qfjnv\") pod \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\" (UID: \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\") "
Sep 08 14:21:02.015522 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.015479    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-dshm\") pod \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\" (UID: \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\") "
Sep 08 14:21:02.015522 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.015510    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-tmp-dir\") pod \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\" (UID: \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\") "
Sep 08 14:21:02.015821 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.015572    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-cabundle-cert\") pod \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\" (UID: \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\") "
Sep 08 14:21:02.015821 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.015641    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-model-cache\") pod \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\" (UID: \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\") "
Sep 08 14:21:02.015821 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.015674    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-kserve-provision-location\") pod \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\" (UID: \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\") "
Sep 08 14:21:02.015821 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.015702    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-home\") pod \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\" (UID: \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\") "
Sep 08 14:21:02.015821 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.015752    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-tls-certs\") pod \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\" (UID: \"4e599e63-cbb3-4528-8d0f-d451eacfb3e6\") "
Sep 08 14:21:02.016093 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.015996    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4e599e63-cbb3-4528-8d0f-d451eacfb3e6" (UID: "4e599e63-cbb3-4528-8d0f-d451eacfb3e6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:21:02.016321 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.016248    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-model-cache" (OuterVolumeSpecName: "model-cache") pod "4e599e63-cbb3-4528-8d0f-d451eacfb3e6" (UID: "4e599e63-cbb3-4528-8d0f-d451eacfb3e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:21:02.016321 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.016265    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-home" (OuterVolumeSpecName: "home") pod "4e599e63-cbb3-4528-8d0f-d451eacfb3e6" (UID: "4e599e63-cbb3-4528-8d0f-d451eacfb3e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:21:02.018327 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.018298    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e599e63-cbb3-4528-8d0f-d451eacfb3e6" (UID: "4e599e63-cbb3-4528-8d0f-d451eacfb3e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:21:02.018445 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.018298    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-dshm" (OuterVolumeSpecName: "dshm") pod "4e599e63-cbb3-4528-8d0f-d451eacfb3e6" (UID: "4e599e63-cbb3-4528-8d0f-d451eacfb3e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:21:02.018445 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.018344    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-kube-api-access-qfjnv" (OuterVolumeSpecName: "kube-api-access-qfjnv") pod "4e599e63-cbb3-4528-8d0f-d451eacfb3e6" (UID: "4e599e63-cbb3-4528-8d0f-d451eacfb3e6"). InnerVolumeSpecName "kube-api-access-qfjnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:21:02.028840 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.028809    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4e599e63-cbb3-4528-8d0f-d451eacfb3e6" (UID: "4e599e63-cbb3-4528-8d0f-d451eacfb3e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:21:02.048080 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.048044    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4e599e63-cbb3-4528-8d0f-d451eacfb3e6" (UID: "4e599e63-cbb3-4528-8d0f-d451eacfb3e6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:21:02.116574 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.116538    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-cabundle-cert\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:21:02.116574 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.116571    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:21:02.116798 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.116584    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-kserve-provision-location\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:21:02.116798 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.116613    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:21:02.116798 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.116626    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:21:02.116798 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.116638    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfjnv\" (UniqueName: \"kubernetes.io/projected/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-kube-api-access-qfjnv\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:21:02.116798 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.116649    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:21:02.116798 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.116661    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e599e63-cbb3-4528-8d0f-d451eacfb3e6-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:21:02.530889 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.530850    2558 generic.go:356] "Generic (PLEG): container finished" podID="4e599e63-cbb3-4528-8d0f-d451eacfb3e6" containerID="8844f90ab250f8c94274f16e8792616d95c9227c5ea731aeafeaa464858a044b" exitCode=0
Sep 08 14:21:02.531279 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.530908    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"
Sep 08 14:21:02.531279 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.530942    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs" event={"ID":"4e599e63-cbb3-4528-8d0f-d451eacfb3e6","Type":"ContainerDied","Data":"8844f90ab250f8c94274f16e8792616d95c9227c5ea731aeafeaa464858a044b"}
Sep 08 14:21:02.531279 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.531012    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs" event={"ID":"4e599e63-cbb3-4528-8d0f-d451eacfb3e6","Type":"ContainerDied","Data":"fcdd9f92d2d8cfa1d830922bbba684e2da81c7aec720acc50fb27220b5f3e3ea"}
Sep 08 14:21:02.531279 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.531050    2558 scope.go:117] "RemoveContainer" containerID="8844f90ab250f8c94274f16e8792616d95c9227c5ea731aeafeaa464858a044b"
Sep 08 14:21:02.541321 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.541299    2558 scope.go:117] "RemoveContainer" containerID="f8cae5a0ae45081d77f8a9a2c39e861acca40c80e8e611982017260b15823428"
Sep 08 14:21:02.550004 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.549979    2558 scope.go:117] "RemoveContainer" containerID="8844f90ab250f8c94274f16e8792616d95c9227c5ea731aeafeaa464858a044b"
Sep 08 14:21:02.550302 ip-10-0-128-147 kubenswrapper[2558]: E0908 14:21:02.550283    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8844f90ab250f8c94274f16e8792616d95c9227c5ea731aeafeaa464858a044b\": container with ID starting with 8844f90ab250f8c94274f16e8792616d95c9227c5ea731aeafeaa464858a044b not found: ID does not exist" containerID="8844f90ab250f8c94274f16e8792616d95c9227c5ea731aeafeaa464858a044b"
Sep 08 14:21:02.550356 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.550312    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8844f90ab250f8c94274f16e8792616d95c9227c5ea731aeafeaa464858a044b"} err="failed to get container status \"8844f90ab250f8c94274f16e8792616d95c9227c5ea731aeafeaa464858a044b\": rpc error: code = NotFound desc = could not find container \"8844f90ab250f8c94274f16e8792616d95c9227c5ea731aeafeaa464858a044b\": container with ID starting with 8844f90ab250f8c94274f16e8792616d95c9227c5ea731aeafeaa464858a044b not found: ID does not exist"
Sep 08 14:21:02.550356 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.550331    2558 scope.go:117] "RemoveContainer" containerID="f8cae5a0ae45081d77f8a9a2c39e861acca40c80e8e611982017260b15823428"
Sep 08 14:21:02.550567 ip-10-0-128-147 kubenswrapper[2558]: E0908 14:21:02.550546    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8cae5a0ae45081d77f8a9a2c39e861acca40c80e8e611982017260b15823428\": container with ID starting with f8cae5a0ae45081d77f8a9a2c39e861acca40c80e8e611982017260b15823428 not found: ID does not exist" containerID="f8cae5a0ae45081d77f8a9a2c39e861acca40c80e8e611982017260b15823428"
Sep 08 14:21:02.550646 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.550579    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8cae5a0ae45081d77f8a9a2c39e861acca40c80e8e611982017260b15823428"} err="failed to get container status \"f8cae5a0ae45081d77f8a9a2c39e861acca40c80e8e611982017260b15823428\": rpc error: code = NotFound desc = could not find container \"f8cae5a0ae45081d77f8a9a2c39e861acca40c80e8e611982017260b15823428\": container with ID starting with f8cae5a0ae45081d77f8a9a2c39e861acca40c80e8e611982017260b15823428 not found: ID does not exist"
Sep 08 14:21:02.555143 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.555118    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"]
Sep 08 14:21:02.559657 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.559635    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-95967666f-qtbxs"]
Sep 08 14:21:02.625520 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:02.625478    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2" podUID="1ff739e2-1ad2-4031-a964-1f55201f5fd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 08 14:21:03.054659 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:03.054624    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e599e63-cbb3-4528-8d0f-d451eacfb3e6" path="/var/lib/kubelet/pods/4e599e63-cbb3-4528-8d0f-d451eacfb3e6/volumes"
Sep 08 14:21:12.625870 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:12.625826    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2" podUID="1ff739e2-1ad2-4031-a964-1f55201f5fd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 08 14:21:20.299581 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.299543    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"]
Sep 08 14:21:20.300125 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.300103    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e599e63-cbb3-4528-8d0f-d451eacfb3e6" containerName="storage-initializer"
Sep 08 14:21:20.300205 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.300129    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e599e63-cbb3-4528-8d0f-d451eacfb3e6" containerName="storage-initializer"
Sep 08 14:21:20.300205 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.300148    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e599e63-cbb3-4528-8d0f-d451eacfb3e6" containerName="main"
Sep 08 14:21:20.300205 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.300157    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e599e63-cbb3-4528-8d0f-d451eacfb3e6" containerName="main"
Sep 08 14:21:20.300365 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.300272    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e599e63-cbb3-4528-8d0f-d451eacfb3e6" containerName="main"
Sep 08 14:21:20.305008 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.304984    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"
Sep 08 14:21:20.307632 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.307584    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 08 14:21:20.307812 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.307787    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-tx2tc\""
Sep 08 14:21:20.307943 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.307660    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 08 14:21:20.307943 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.307900    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 08 14:21:20.307943 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.307606    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Sep 08 14:21:20.308199 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.308179    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 14:21:20.320613 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.320567    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"]
Sep 08 14:21:20.378564 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.378503    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ba7426f-c99d-4b74-a19e-9895720a156c-kserve-provision-location\") pod \"tls-verification-test-kserve-5686ffb94f-w4wrk\" (UID: \"7ba7426f-c99d-4b74-a19e-9895720a156c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"
Sep 08 14:21:20.378801 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.378662    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ba7426f-c99d-4b74-a19e-9895720a156c-dshm\") pod \"tls-verification-test-kserve-5686ffb94f-w4wrk\" (UID: \"7ba7426f-c99d-4b74-a19e-9895720a156c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"
Sep 08 14:21:20.378801 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.378745    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prwsp\" (UniqueName: \"kubernetes.io/projected/7ba7426f-c99d-4b74-a19e-9895720a156c-kube-api-access-prwsp\") pod \"tls-verification-test-kserve-5686ffb94f-w4wrk\" (UID: \"7ba7426f-c99d-4b74-a19e-9895720a156c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"
Sep 08 14:21:20.378924 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.378816    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ba7426f-c99d-4b74-a19e-9895720a156c-tls-certs\") pod \"tls-verification-test-kserve-5686ffb94f-w4wrk\" (UID: \"7ba7426f-c99d-4b74-a19e-9895720a156c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"
Sep 08 14:21:20.378982 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.378926    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ba7426f-c99d-4b74-a19e-9895720a156c-cabundle-cert\") pod \"tls-verification-test-kserve-5686ffb94f-w4wrk\" (UID: \"7ba7426f-c99d-4b74-a19e-9895720a156c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"
Sep 08 14:21:20.379023 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.379004    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ba7426f-c99d-4b74-a19e-9895720a156c-home\") pod \"tls-verification-test-kserve-5686ffb94f-w4wrk\" (UID: \"7ba7426f-c99d-4b74-a19e-9895720a156c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"
Sep 08 14:21:20.379064 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.379041    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ba7426f-c99d-4b74-a19e-9895720a156c-model-cache\") pod \"tls-verification-test-kserve-5686ffb94f-w4wrk\" (UID: \"7ba7426f-c99d-4b74-a19e-9895720a156c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"
Sep 08 14:21:20.379100 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.379061    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ba7426f-c99d-4b74-a19e-9895720a156c-tmp-dir\") pod \"tls-verification-test-kserve-5686ffb94f-w4wrk\" (UID: \"7ba7426f-c99d-4b74-a19e-9895720a156c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"
Sep 08 14:21:20.480571 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.480527    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ba7426f-c99d-4b74-a19e-9895720a156c-dshm\") pod \"tls-verification-test-kserve-5686ffb94f-w4wrk\" (UID: \"7ba7426f-c99d-4b74-a19e-9895720a156c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"
Sep 08 14:21:20.480730 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.480619    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prwsp\" (UniqueName: \"kubernetes.io/projected/7ba7426f-c99d-4b74-a19e-9895720a156c-kube-api-access-prwsp\") pod \"tls-verification-test-kserve-5686ffb94f-w4wrk\" (UID: \"7ba7426f-c99d-4b74-a19e-9895720a156c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"
Sep 08 14:21:20.480730 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.480670    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ba7426f-c99d-4b74-a19e-9895720a156c-tls-certs\") pod \"tls-verification-test-kserve-5686ffb94f-w4wrk\" (UID: \"7ba7426f-c99d-4b74-a19e-9895720a156c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"
Sep 08 14:21:20.480844 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.480730    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ba7426f-c99d-4b74-a19e-9895720a156c-cabundle-cert\") pod \"tls-verification-test-kserve-5686ffb94f-w4wrk\" (UID: \"7ba7426f-c99d-4b74-a19e-9895720a156c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"
Sep 08 14:21:20.480844 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.480778    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ba7426f-c99d-4b74-a19e-9895720a156c-home\") pod \"tls-verification-test-kserve-5686ffb94f-w4wrk\" (UID: \"7ba7426f-c99d-4b74-a19e-9895720a156c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"
Sep 08 14:21:20.480844 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.480818    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ba7426f-c99d-4b74-a19e-9895720a156c-model-cache\") pod \"tls-verification-test-kserve-5686ffb94f-w4wrk\" (UID: \"7ba7426f-c99d-4b74-a19e-9895720a156c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"
Sep 08 14:21:20.480994 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.480841    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ba7426f-c99d-4b74-a19e-9895720a156c-tmp-dir\") pod \"tls-verification-test-kserve-5686ffb94f-w4wrk\" (UID: \"7ba7426f-c99d-4b74-a19e-9895720a156c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"
Sep 08 14:21:20.480994 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.480910    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ba7426f-c99d-4b74-a19e-9895720a156c-kserve-provision-location\") pod \"tls-verification-test-kserve-5686ffb94f-w4wrk\" (UID: \"7ba7426f-c99d-4b74-a19e-9895720a156c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"
Sep 08 14:21:20.481235 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.481207    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ba7426f-c99d-4b74-a19e-9895720a156c-home\") pod \"tls-verification-test-kserve-5686ffb94f-w4wrk\" (UID: \"7ba7426f-c99d-4b74-a19e-9895720a156c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"
Sep 08 14:21:20.481427 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.481287    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ba7426f-c99d-4b74-a19e-9895720a156c-kserve-provision-location\") pod \"tls-verification-test-kserve-5686ffb94f-w4wrk\" (UID: \"7ba7426f-c99d-4b74-a19e-9895720a156c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"
Sep 08 14:21:20.481427 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.481309    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ba7426f-c99d-4b74-a19e-9895720a156c-model-cache\") pod \"tls-verification-test-kserve-5686ffb94f-w4wrk\" (UID: \"7ba7426f-c99d-4b74-a19e-9895720a156c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"
Sep 08 14:21:20.481630 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.481538    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ba7426f-c99d-4b74-a19e-9895720a156c-cabundle-cert\") pod \"tls-verification-test-kserve-5686ffb94f-w4wrk\" (UID: \"7ba7426f-c99d-4b74-a19e-9895720a156c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"
Sep 08 14:21:20.481630 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.481549    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ba7426f-c99d-4b74-a19e-9895720a156c-tmp-dir\") pod \"tls-verification-test-kserve-5686ffb94f-w4wrk\" (UID: \"7ba7426f-c99d-4b74-a19e-9895720a156c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"
Sep 08 14:21:20.482944 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.482920    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ba7426f-c99d-4b74-a19e-9895720a156c-dshm\") pod \"tls-verification-test-kserve-5686ffb94f-w4wrk\" (UID: \"7ba7426f-c99d-4b74-a19e-9895720a156c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"
Sep 08 14:21:20.483452 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.483429    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ba7426f-c99d-4b74-a19e-9895720a156c-tls-certs\") pod \"tls-verification-test-kserve-5686ffb94f-w4wrk\" (UID: \"7ba7426f-c99d-4b74-a19e-9895720a156c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"
Sep 08 14:21:20.489548 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.489522    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prwsp\" (UniqueName: \"kubernetes.io/projected/7ba7426f-c99d-4b74-a19e-9895720a156c-kube-api-access-prwsp\") pod \"tls-verification-test-kserve-5686ffb94f-w4wrk\" (UID: \"7ba7426f-c99d-4b74-a19e-9895720a156c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"
Sep 08 14:21:20.616548 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.616513    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"
Sep 08 14:21:20.903528 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:20.903495    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"]
Sep 08 14:21:20.905159 ip-10-0-128-147 kubenswrapper[2558]: W0908 14:21:20.905121    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba7426f_c99d_4b74_a19e_9895720a156c.slice/crio-1c38ce9602983b0f8d96780cc616942cb6b82a7a1b3cb5c821289b3285345e00 WatchSource:0}: Error finding container 1c38ce9602983b0f8d96780cc616942cb6b82a7a1b3cb5c821289b3285345e00: Status 404 returned error can't find the container with id 1c38ce9602983b0f8d96780cc616942cb6b82a7a1b3cb5c821289b3285345e00
Sep 08 14:21:21.632339 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:21.632302    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk" event={"ID":"7ba7426f-c99d-4b74-a19e-9895720a156c","Type":"ContainerStarted","Data":"288ac4217b90e6208146acd3a6db7b99212afbfa0fbb91a87ba2390f06c3d02d"}
Sep 08 14:21:21.632773 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:21.632365    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk" event={"ID":"7ba7426f-c99d-4b74-a19e-9895720a156c","Type":"ContainerStarted","Data":"1c38ce9602983b0f8d96780cc616942cb6b82a7a1b3cb5c821289b3285345e00"}
Sep 08 14:21:22.626132 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:22.626084    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2" podUID="1ff739e2-1ad2-4031-a964-1f55201f5fd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 08 14:21:23.646275 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:23.646236    2558 generic.go:356] "Generic (PLEG): container finished" podID="7ba7426f-c99d-4b74-a19e-9895720a156c" containerID="288ac4217b90e6208146acd3a6db7b99212afbfa0fbb91a87ba2390f06c3d02d" exitCode=0
Sep 08 14:21:23.646732 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:23.646311    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk" event={"ID":"7ba7426f-c99d-4b74-a19e-9895720a156c","Type":"ContainerDied","Data":"288ac4217b90e6208146acd3a6db7b99212afbfa0fbb91a87ba2390f06c3d02d"}
Sep 08 14:21:24.654404 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:24.654364    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk" event={"ID":"7ba7426f-c99d-4b74-a19e-9895720a156c","Type":"ContainerStarted","Data":"5cb05e7e360f2ea7aa92cd32d12afdc49fe907ded30f78e89e3b6cfc8f39ec50"}
Sep 08 14:21:24.683338 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:24.683262    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk" podStartSLOduration=2.310588439 podStartE2EDuration="4.683237952s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.372649513s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:21:24.680765034 +0000 UTC m=+3948.132525967" watchObservedRunningTime="2026-09-08 14:21:24.683237952 +0000 UTC m=+3948.134998883"
Sep 08 14:21:25.617284 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:25.617242    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"
Sep 08 14:21:30.617236 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:30.617138    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"
Sep 08 14:21:30.618843 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:30.618804    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk" podUID="7ba7426f-c99d-4b74-a19e-9895720a156c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Sep 08 14:21:32.626001 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:32.625954    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2" podUID="1ff739e2-1ad2-4031-a964-1f55201f5fd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 08 14:21:40.618130 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:40.618076    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk" podUID="7ba7426f-c99d-4b74-a19e-9895720a156c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Sep 08 14:21:42.625617 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:42.625554    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2" podUID="1ff739e2-1ad2-4031-a964-1f55201f5fd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 08 14:21:50.617454 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:50.617401    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk" podUID="7ba7426f-c99d-4b74-a19e-9895720a156c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Sep 08 14:21:52.626607 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:21:52.626541    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2" podUID="1ff739e2-1ad2-4031-a964-1f55201f5fd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 08 14:22:00.617257 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:00.617200    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk" podUID="7ba7426f-c99d-4b74-a19e-9895720a156c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Sep 08 14:22:02.625575 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:02.625527    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2" podUID="1ff739e2-1ad2-4031-a964-1f55201f5fd5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 08 14:22:10.617745 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:10.617694    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk" podUID="7ba7426f-c99d-4b74-a19e-9895720a156c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Sep 08 14:22:12.636576 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:12.636538    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"
Sep 08 14:22:12.645540 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:12.645515    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"
Sep 08 14:22:20.617138 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:20.617082    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk" podUID="7ba7426f-c99d-4b74-a19e-9895720a156c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Sep 08 14:22:21.673953 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:21.673917    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"]
Sep 08 14:22:30.617650 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:30.617585    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk" podUID="7ba7426f-c99d-4b74-a19e-9895720a156c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Sep 08 14:22:36.698088 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:36.698009    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2" podUID="1ff739e2-1ad2-4031-a964-1f55201f5fd5" containerName="main" containerID="cri-o://f133cf7f37b4921f4ad72dbb954041af4259c64c05424794406048aefed6be98" gracePeriod=45
Sep 08 14:22:37.454245 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:37.454215    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"
Sep 08 14:22:37.520219 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:37.520137    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ff739e2-1ad2-4031-a964-1f55201f5fd5-model-cache\") pod \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\" (UID: \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\") "
Sep 08 14:22:37.520219 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:37.520184    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bzxw\" (UniqueName: \"kubernetes.io/projected/1ff739e2-1ad2-4031-a964-1f55201f5fd5-kube-api-access-5bzxw\") pod \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\" (UID: \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\") "
Sep 08 14:22:37.520444 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:37.520231    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ff739e2-1ad2-4031-a964-1f55201f5fd5-tls-certs\") pod \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\" (UID: \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\") "
Sep 08 14:22:37.520444 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:37.520263    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ff739e2-1ad2-4031-a964-1f55201f5fd5-tmp-dir\") pod \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\" (UID: \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\") "
Sep 08 14:22:37.520444 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:37.520298    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ff739e2-1ad2-4031-a964-1f55201f5fd5-dshm\") pod \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\" (UID: \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\") "
Sep 08 14:22:37.520444 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:37.520319    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ff739e2-1ad2-4031-a964-1f55201f5fd5-cabundle-cert\") pod \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\" (UID: \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\") "
Sep 08 14:22:37.520444 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:37.520364    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ff739e2-1ad2-4031-a964-1f55201f5fd5-kserve-provision-location\") pod \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\" (UID: \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\") "
Sep 08 14:22:37.520444 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:37.520389    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ff739e2-1ad2-4031-a964-1f55201f5fd5-home\") pod \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\" (UID: \"1ff739e2-1ad2-4031-a964-1f55201f5fd5\") "
Sep 08 14:22:37.520808 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:37.520464    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff739e2-1ad2-4031-a964-1f55201f5fd5-model-cache" (OuterVolumeSpecName: "model-cache") pod "1ff739e2-1ad2-4031-a964-1f55201f5fd5" (UID: "1ff739e2-1ad2-4031-a964-1f55201f5fd5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:22:37.520808 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:37.520718    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ff739e2-1ad2-4031-a964-1f55201f5fd5-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:22:37.522475 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:37.520935    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ff739e2-1ad2-4031-a964-1f55201f5fd5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1ff739e2-1ad2-4031-a964-1f55201f5fd5" (UID: "1ff739e2-1ad2-4031-a964-1f55201f5fd5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:22:37.522475 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:37.521325    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff739e2-1ad2-4031-a964-1f55201f5fd5-home" (OuterVolumeSpecName: "home") pod "1ff739e2-1ad2-4031-a964-1f55201f5fd5" (UID: "1ff739e2-1ad2-4031-a964-1f55201f5fd5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:22:37.522727 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:37.522698    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ff739e2-1ad2-4031-a964-1f55201f5fd5-kube-api-access-5bzxw" (OuterVolumeSpecName: "kube-api-access-5bzxw") pod "1ff739e2-1ad2-4031-a964-1f55201f5fd5" (UID: "1ff739e2-1ad2-4031-a964-1f55201f5fd5"). InnerVolumeSpecName "kube-api-access-5bzxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:22:37.523138 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:37.523110    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ff739e2-1ad2-4031-a964-1f55201f5fd5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1ff739e2-1ad2-4031-a964-1f55201f5fd5" (UID: "1ff739e2-1ad2-4031-a964-1f55201f5fd5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:22:37.523407 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:37.523387    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff739e2-1ad2-4031-a964-1f55201f5fd5-dshm" (OuterVolumeSpecName: "dshm") pod "1ff739e2-1ad2-4031-a964-1f55201f5fd5" (UID: "1ff739e2-1ad2-4031-a964-1f55201f5fd5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:22:37.533187 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:37.533150    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff739e2-1ad2-4031-a964-1f55201f5fd5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1ff739e2-1ad2-4031-a964-1f55201f5fd5" (UID: "1ff739e2-1ad2-4031-a964-1f55201f5fd5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:22:37.543640 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:37.543546    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ff739e2-1ad2-4031-a964-1f55201f5fd5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1ff739e2-1ad2-4031-a964-1f55201f5fd5" (UID: "1ff739e2-1ad2-4031-a964-1f55201f5fd5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:22:37.621370 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:37.621308    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ff739e2-1ad2-4031-a964-1f55201f5fd5-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:22:37.621370 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:37.621350    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1ff739e2-1ad2-4031-a964-1f55201f5fd5-cabundle-cert\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:22:37.621370 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:37.621361    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ff739e2-1ad2-4031-a964-1f55201f5fd5-kserve-provision-location\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:22:37.621370 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:37.621370    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ff739e2-1ad2-4031-a964-1f55201f5fd5-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:22:37.621370 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:37.621378    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bzxw\" (UniqueName: \"kubernetes.io/projected/1ff739e2-1ad2-4031-a964-1f55201f5fd5-kube-api-access-5bzxw\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:22:37.621370 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:37.621387    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ff739e2-1ad2-4031-a964-1f55201f5fd5-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:22:37.621370 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:37.621395    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ff739e2-1ad2-4031-a964-1f55201f5fd5-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:22:38.061924 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:38.061882    2558 generic.go:356] "Generic (PLEG): container finished" podID="1ff739e2-1ad2-4031-a964-1f55201f5fd5" containerID="f133cf7f37b4921f4ad72dbb954041af4259c64c05424794406048aefed6be98" exitCode=0
Sep 08 14:22:38.062409 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:38.061939    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"
Sep 08 14:22:38.062409 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:38.061960    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2" event={"ID":"1ff739e2-1ad2-4031-a964-1f55201f5fd5","Type":"ContainerDied","Data":"f133cf7f37b4921f4ad72dbb954041af4259c64c05424794406048aefed6be98"}
Sep 08 14:22:38.062409 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:38.062015    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2" event={"ID":"1ff739e2-1ad2-4031-a964-1f55201f5fd5","Type":"ContainerDied","Data":"2445d6e5a291552ad0c60c812b7ea736d585290ae122a0ce21e6f974ac779f0d"}
Sep 08 14:22:38.062409 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:38.062037    2558 scope.go:117] "RemoveContainer" containerID="f133cf7f37b4921f4ad72dbb954041af4259c64c05424794406048aefed6be98"
Sep 08 14:22:38.072975 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:38.072949    2558 scope.go:117] "RemoveContainer" containerID="dbd7b79b254f7e13860bfd2ea26050a3e41a92cde0ff8f620c94a62ee995496d"
Sep 08 14:22:38.081320 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:38.081299    2558 scope.go:117] "RemoveContainer" containerID="f133cf7f37b4921f4ad72dbb954041af4259c64c05424794406048aefed6be98"
Sep 08 14:22:38.081640 ip-10-0-128-147 kubenswrapper[2558]: E0908 14:22:38.081619    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f133cf7f37b4921f4ad72dbb954041af4259c64c05424794406048aefed6be98\": container with ID starting with f133cf7f37b4921f4ad72dbb954041af4259c64c05424794406048aefed6be98 not found: ID does not exist" containerID="f133cf7f37b4921f4ad72dbb954041af4259c64c05424794406048aefed6be98"
Sep 08 14:22:38.081741 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:38.081650    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f133cf7f37b4921f4ad72dbb954041af4259c64c05424794406048aefed6be98"} err="failed to get container status \"f133cf7f37b4921f4ad72dbb954041af4259c64c05424794406048aefed6be98\": rpc error: code = NotFound desc = could not find container \"f133cf7f37b4921f4ad72dbb954041af4259c64c05424794406048aefed6be98\": container with ID starting with f133cf7f37b4921f4ad72dbb954041af4259c64c05424794406048aefed6be98 not found: ID does not exist"
Sep 08 14:22:38.081741 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:38.081668    2558 scope.go:117] "RemoveContainer" containerID="dbd7b79b254f7e13860bfd2ea26050a3e41a92cde0ff8f620c94a62ee995496d"
Sep 08 14:22:38.081926 ip-10-0-128-147 kubenswrapper[2558]: E0908 14:22:38.081906    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbd7b79b254f7e13860bfd2ea26050a3e41a92cde0ff8f620c94a62ee995496d\": container with ID starting with dbd7b79b254f7e13860bfd2ea26050a3e41a92cde0ff8f620c94a62ee995496d not found: ID does not exist" containerID="dbd7b79b254f7e13860bfd2ea26050a3e41a92cde0ff8f620c94a62ee995496d"
Sep 08 14:22:38.081988 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:38.081931    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbd7b79b254f7e13860bfd2ea26050a3e41a92cde0ff8f620c94a62ee995496d"} err="failed to get container status \"dbd7b79b254f7e13860bfd2ea26050a3e41a92cde0ff8f620c94a62ee995496d\": rpc error: code = NotFound desc = could not find container \"dbd7b79b254f7e13860bfd2ea26050a3e41a92cde0ff8f620c94a62ee995496d\": container with ID starting with dbd7b79b254f7e13860bfd2ea26050a3e41a92cde0ff8f620c94a62ee995496d not found: ID does not exist"
Sep 08 14:22:38.086675 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:38.086646    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"]
Sep 08 14:22:38.091661 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:38.091634    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-66bd774bc4-4shp2"]
Sep 08 14:22:39.054611 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:39.054552    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ff739e2-1ad2-4031-a964-1f55201f5fd5" path="/var/lib/kubelet/pods/1ff739e2-1ad2-4031-a964-1f55201f5fd5/volumes"
Sep 08 14:22:40.617237 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:40.617190    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk" podUID="7ba7426f-c99d-4b74-a19e-9895720a156c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Sep 08 14:22:50.617943 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:22:50.617897    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk" podUID="7ba7426f-c99d-4b74-a19e-9895720a156c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Sep 08 14:23:00.617788 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:00.617691    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk" podUID="7ba7426f-c99d-4b74-a19e-9895720a156c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Sep 08 14:23:10.617099 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:10.617052    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk" podUID="7ba7426f-c99d-4b74-a19e-9895720a156c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Sep 08 14:23:20.627249 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:20.627213    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"
Sep 08 14:23:20.635165 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:20.635134    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"
Sep 08 14:23:30.890170 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:30.890134    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"]
Sep 08 14:23:30.890672 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:30.890576    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ff739e2-1ad2-4031-a964-1f55201f5fd5" containerName="main"
Sep 08 14:23:30.890672 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:30.890606    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ff739e2-1ad2-4031-a964-1f55201f5fd5" containerName="main"
Sep 08 14:23:30.890672 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:30.890629    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ff739e2-1ad2-4031-a964-1f55201f5fd5" containerName="storage-initializer"
Sep 08 14:23:30.890672 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:30.890635    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ff739e2-1ad2-4031-a964-1f55201f5fd5" containerName="storage-initializer"
Sep 08 14:23:30.890813 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:30.890706    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ff739e2-1ad2-4031-a964-1f55201f5fd5" containerName="main"
Sep 08 14:23:30.893789 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:30.893767    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"
Sep 08 14:23:30.896178 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:30.896142    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 08 14:23:30.896381 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:30.896192    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 08 14:23:30.896381 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:30.896321    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 08 14:23:30.896675 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:30.896651    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 14:23:30.896777 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:30.896682    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-m98fm\""
Sep 08 14:23:30.896961 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:30.896942    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 08 14:23:30.905812 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:30.905790    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"]
Sep 08 14:23:30.994130 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:30.994085    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eb596d2-d241-4e46-a4eb-b33af3a5011f-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6\" (UID: \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"
Sep 08 14:23:30.994310 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:30.994162    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eb596d2-d241-4e46-a4eb-b33af3a5011f-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6\" (UID: \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"
Sep 08 14:23:30.994310 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:30.994234    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eb596d2-d241-4e46-a4eb-b33af3a5011f-home\") pod \"router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6\" (UID: \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"
Sep 08 14:23:30.994310 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:30.994258    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4eb596d2-d241-4e46-a4eb-b33af3a5011f-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6\" (UID: \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"
Sep 08 14:23:30.994310 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:30.994283    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eb596d2-d241-4e46-a4eb-b33af3a5011f-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6\" (UID: \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"
Sep 08 14:23:30.994310 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:30.994307    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4eb596d2-d241-4e46-a4eb-b33af3a5011f-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6\" (UID: \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"
Sep 08 14:23:30.994488 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:30.994328    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ntgb\" (UniqueName: \"kubernetes.io/projected/4eb596d2-d241-4e46-a4eb-b33af3a5011f-kube-api-access-4ntgb\") pod \"router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6\" (UID: \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"
Sep 08 14:23:30.994488 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:30.994396    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eb596d2-d241-4e46-a4eb-b33af3a5011f-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6\" (UID: \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"
Sep 08 14:23:31.095868 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:31.095822    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eb596d2-d241-4e46-a4eb-b33af3a5011f-home\") pod \"router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6\" (UID: \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"
Sep 08 14:23:31.095868 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:31.095871    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4eb596d2-d241-4e46-a4eb-b33af3a5011f-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6\" (UID: \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"
Sep 08 14:23:31.096138 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:31.095908    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eb596d2-d241-4e46-a4eb-b33af3a5011f-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6\" (UID: \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"
Sep 08 14:23:31.096138 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:31.095931    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4eb596d2-d241-4e46-a4eb-b33af3a5011f-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6\" (UID: \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"
Sep 08 14:23:31.096138 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:31.095954    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ntgb\" (UniqueName: \"kubernetes.io/projected/4eb596d2-d241-4e46-a4eb-b33af3a5011f-kube-api-access-4ntgb\") pod \"router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6\" (UID: \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"
Sep 08 14:23:31.096138 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:31.095984    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eb596d2-d241-4e46-a4eb-b33af3a5011f-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6\" (UID: \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"
Sep 08 14:23:31.096138 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:31.096007    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eb596d2-d241-4e46-a4eb-b33af3a5011f-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6\" (UID: \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"
Sep 08 14:23:31.096138 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:31.096065    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eb596d2-d241-4e46-a4eb-b33af3a5011f-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6\" (UID: \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"
Sep 08 14:23:31.096468 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:31.096310    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eb596d2-d241-4e46-a4eb-b33af3a5011f-home\") pod \"router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6\" (UID: \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"
Sep 08 14:23:31.096468 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:31.096341    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eb596d2-d241-4e46-a4eb-b33af3a5011f-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6\" (UID: \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"
Sep 08 14:23:31.096571 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:31.096481    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4eb596d2-d241-4e46-a4eb-b33af3a5011f-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6\" (UID: \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"
Sep 08 14:23:31.096648 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:31.096631    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eb596d2-d241-4e46-a4eb-b33af3a5011f-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6\" (UID: \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"
Sep 08 14:23:31.096819 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:31.096793    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4eb596d2-d241-4e46-a4eb-b33af3a5011f-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6\" (UID: \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"
Sep 08 14:23:31.098432 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:31.098409    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eb596d2-d241-4e46-a4eb-b33af3a5011f-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6\" (UID: \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"
Sep 08 14:23:31.098615 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:31.098580    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eb596d2-d241-4e46-a4eb-b33af3a5011f-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6\" (UID: \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"
Sep 08 14:23:31.103582 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:31.103551    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ntgb\" (UniqueName: \"kubernetes.io/projected/4eb596d2-d241-4e46-a4eb-b33af3a5011f-kube-api-access-4ntgb\") pod \"router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6\" (UID: \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"
Sep 08 14:23:31.205904 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:31.205807    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"
Sep 08 14:23:31.495711 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:31.495617    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"]
Sep 08 14:23:31.499049 ip-10-0-128-147 kubenswrapper[2558]: W0908 14:23:31.499022    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb596d2_d241_4e46_a4eb_b33af3a5011f.slice/crio-6a86e66c4b49139beb54acf0f463e5142854fb61e5ec33112dc524cad74ee20c WatchSource:0}: Error finding container 6a86e66c4b49139beb54acf0f463e5142854fb61e5ec33112dc524cad74ee20c: Status 404 returned error can't find the container with id 6a86e66c4b49139beb54acf0f463e5142854fb61e5ec33112dc524cad74ee20c
Sep 08 14:23:32.332264 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:32.332220    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6" event={"ID":"4eb596d2-d241-4e46-a4eb-b33af3a5011f","Type":"ContainerStarted","Data":"f51ff2ba1e757e73f89c0f9fbbcb9311d886529f97b9b8fe5d027ebc54e8bdb2"}
Sep 08 14:23:32.332676 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:32.332294    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6" event={"ID":"4eb596d2-d241-4e46-a4eb-b33af3a5011f","Type":"ContainerStarted","Data":"6a86e66c4b49139beb54acf0f463e5142854fb61e5ec33112dc524cad74ee20c"}
Sep 08 14:23:34.345075 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:34.345037    2558 generic.go:356] "Generic (PLEG): container finished" podID="4eb596d2-d241-4e46-a4eb-b33af3a5011f" containerID="f51ff2ba1e757e73f89c0f9fbbcb9311d886529f97b9b8fe5d027ebc54e8bdb2" exitCode=0
Sep 08 14:23:34.345075 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:34.345071    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6" event={"ID":"4eb596d2-d241-4e46-a4eb-b33af3a5011f","Type":"ContainerDied","Data":"f51ff2ba1e757e73f89c0f9fbbcb9311d886529f97b9b8fe5d027ebc54e8bdb2"}
Sep 08 14:23:35.353420 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:35.353378    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6" event={"ID":"4eb596d2-d241-4e46-a4eb-b33af3a5011f","Type":"ContainerStarted","Data":"6661db98cd82856e80c8c1acdc33aeb204b3f5790697c1d61c8b74ac067fc3e9"}
Sep 08 14:23:35.377395 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:35.377319    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6" podStartSLOduration=3.106116728 podStartE2EDuration="5.377302051s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.271185323s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:23:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:23:35.376134168 +0000 UTC m=+4078.827895103" watchObservedRunningTime="2026-09-08 14:23:35.377302051 +0000 UTC m=+4078.829062997"
Sep 08 14:23:41.206414 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:41.206371    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"
Sep 08 14:23:41.207071 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:41.206448    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"
Sep 08 14:23:41.207871 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:41.207838    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6" podUID="4eb596d2-d241-4e46-a4eb-b33af3a5011f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 08 14:23:48.627607 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:48.627543    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"]
Sep 08 14:23:51.206972 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:23:51.206913    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6" podUID="4eb596d2-d241-4e46-a4eb-b33af3a5011f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 08 14:24:01.206742 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:01.206695    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6" podUID="4eb596d2-d241-4e46-a4eb-b33af3a5011f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 08 14:24:03.652284 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:03.652240    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk" podUID="7ba7426f-c99d-4b74-a19e-9895720a156c" containerName="main" containerID="cri-o://5cb05e7e360f2ea7aa92cd32d12afdc49fe907ded30f78e89e3b6cfc8f39ec50" gracePeriod=45
Sep 08 14:24:04.406375 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.406347    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"
Sep 08 14:24:04.491511 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.491409    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ba7426f-c99d-4b74-a19e-9895720a156c-kserve-provision-location\") pod \"7ba7426f-c99d-4b74-a19e-9895720a156c\" (UID: \"7ba7426f-c99d-4b74-a19e-9895720a156c\") "
Sep 08 14:24:04.491511 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.491459    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ba7426f-c99d-4b74-a19e-9895720a156c-home\") pod \"7ba7426f-c99d-4b74-a19e-9895720a156c\" (UID: \"7ba7426f-c99d-4b74-a19e-9895720a156c\") "
Sep 08 14:24:04.491511 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.491502    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ba7426f-c99d-4b74-a19e-9895720a156c-cabundle-cert\") pod \"7ba7426f-c99d-4b74-a19e-9895720a156c\" (UID: \"7ba7426f-c99d-4b74-a19e-9895720a156c\") "
Sep 08 14:24:04.491841 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.491525    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ba7426f-c99d-4b74-a19e-9895720a156c-tmp-dir\") pod \"7ba7426f-c99d-4b74-a19e-9895720a156c\" (UID: \"7ba7426f-c99d-4b74-a19e-9895720a156c\") "
Sep 08 14:24:04.491841 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.491572    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prwsp\" (UniqueName: \"kubernetes.io/projected/7ba7426f-c99d-4b74-a19e-9895720a156c-kube-api-access-prwsp\") pod \"7ba7426f-c99d-4b74-a19e-9895720a156c\" (UID: \"7ba7426f-c99d-4b74-a19e-9895720a156c\") "
Sep 08 14:24:04.491841 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.491632    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ba7426f-c99d-4b74-a19e-9895720a156c-dshm\") pod \"7ba7426f-c99d-4b74-a19e-9895720a156c\" (UID: \"7ba7426f-c99d-4b74-a19e-9895720a156c\") "
Sep 08 14:24:04.491841 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.491663    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ba7426f-c99d-4b74-a19e-9895720a156c-tls-certs\") pod \"7ba7426f-c99d-4b74-a19e-9895720a156c\" (UID: \"7ba7426f-c99d-4b74-a19e-9895720a156c\") "
Sep 08 14:24:04.491841 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.491716    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ba7426f-c99d-4b74-a19e-9895720a156c-model-cache\") pod \"7ba7426f-c99d-4b74-a19e-9895720a156c\" (UID: \"7ba7426f-c99d-4b74-a19e-9895720a156c\") "
Sep 08 14:24:04.492235 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.492205    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ba7426f-c99d-4b74-a19e-9895720a156c-model-cache" (OuterVolumeSpecName: "model-cache") pod "7ba7426f-c99d-4b74-a19e-9895720a156c" (UID: "7ba7426f-c99d-4b74-a19e-9895720a156c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:24:04.492332 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.492301    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ba7426f-c99d-4b74-a19e-9895720a156c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7ba7426f-c99d-4b74-a19e-9895720a156c" (UID: "7ba7426f-c99d-4b74-a19e-9895720a156c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:24:04.492588 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.492547    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ba7426f-c99d-4b74-a19e-9895720a156c-home" (OuterVolumeSpecName: "home") pod "7ba7426f-c99d-4b74-a19e-9895720a156c" (UID: "7ba7426f-c99d-4b74-a19e-9895720a156c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:24:04.494306 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.494276    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ba7426f-c99d-4b74-a19e-9895720a156c-dshm" (OuterVolumeSpecName: "dshm") pod "7ba7426f-c99d-4b74-a19e-9895720a156c" (UID: "7ba7426f-c99d-4b74-a19e-9895720a156c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:24:04.494572 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.494555    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ba7426f-c99d-4b74-a19e-9895720a156c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7ba7426f-c99d-4b74-a19e-9895720a156c" (UID: "7ba7426f-c99d-4b74-a19e-9895720a156c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:24:04.494763 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.494743    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ba7426f-c99d-4b74-a19e-9895720a156c-kube-api-access-prwsp" (OuterVolumeSpecName: "kube-api-access-prwsp") pod "7ba7426f-c99d-4b74-a19e-9895720a156c" (UID: "7ba7426f-c99d-4b74-a19e-9895720a156c"). InnerVolumeSpecName "kube-api-access-prwsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:24:04.505911 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.505880    2558 generic.go:356] "Generic (PLEG): container finished" podID="7ba7426f-c99d-4b74-a19e-9895720a156c" containerID="5cb05e7e360f2ea7aa92cd32d12afdc49fe907ded30f78e89e3b6cfc8f39ec50" exitCode=0
Sep 08 14:24:04.506020 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.505932    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"
Sep 08 14:24:04.506020 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.505955    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk" event={"ID":"7ba7426f-c99d-4b74-a19e-9895720a156c","Type":"ContainerDied","Data":"5cb05e7e360f2ea7aa92cd32d12afdc49fe907ded30f78e89e3b6cfc8f39ec50"}
Sep 08 14:24:04.506020 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.506014    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk" event={"ID":"7ba7426f-c99d-4b74-a19e-9895720a156c","Type":"ContainerDied","Data":"1c38ce9602983b0f8d96780cc616942cb6b82a7a1b3cb5c821289b3285345e00"}
Sep 08 14:24:04.506179 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.506030    2558 scope.go:117] "RemoveContainer" containerID="5cb05e7e360f2ea7aa92cd32d12afdc49fe907ded30f78e89e3b6cfc8f39ec50"
Sep 08 14:24:04.510364 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.510304    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ba7426f-c99d-4b74-a19e-9895720a156c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7ba7426f-c99d-4b74-a19e-9895720a156c" (UID: "7ba7426f-c99d-4b74-a19e-9895720a156c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:24:04.519929 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.519908    2558 scope.go:117] "RemoveContainer" containerID="288ac4217b90e6208146acd3a6db7b99212afbfa0fbb91a87ba2390f06c3d02d"
Sep 08 14:24:04.522285 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.522260    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ba7426f-c99d-4b74-a19e-9895720a156c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7ba7426f-c99d-4b74-a19e-9895720a156c" (UID: "7ba7426f-c99d-4b74-a19e-9895720a156c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:24:04.527767 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.527740    2558 scope.go:117] "RemoveContainer" containerID="5cb05e7e360f2ea7aa92cd32d12afdc49fe907ded30f78e89e3b6cfc8f39ec50"
Sep 08 14:24:04.528065 ip-10-0-128-147 kubenswrapper[2558]: E0908 14:24:04.528044    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cb05e7e360f2ea7aa92cd32d12afdc49fe907ded30f78e89e3b6cfc8f39ec50\": container with ID starting with 5cb05e7e360f2ea7aa92cd32d12afdc49fe907ded30f78e89e3b6cfc8f39ec50 not found: ID does not exist" containerID="5cb05e7e360f2ea7aa92cd32d12afdc49fe907ded30f78e89e3b6cfc8f39ec50"
Sep 08 14:24:04.528147 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.528074    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cb05e7e360f2ea7aa92cd32d12afdc49fe907ded30f78e89e3b6cfc8f39ec50"} err="failed to get container status \"5cb05e7e360f2ea7aa92cd32d12afdc49fe907ded30f78e89e3b6cfc8f39ec50\": rpc error: code = NotFound desc = could not find container \"5cb05e7e360f2ea7aa92cd32d12afdc49fe907ded30f78e89e3b6cfc8f39ec50\": container with ID starting with 5cb05e7e360f2ea7aa92cd32d12afdc49fe907ded30f78e89e3b6cfc8f39ec50 not found: ID does not exist"
Sep 08 14:24:04.528147 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.528094    2558 scope.go:117] "RemoveContainer" containerID="288ac4217b90e6208146acd3a6db7b99212afbfa0fbb91a87ba2390f06c3d02d"
Sep 08 14:24:04.528363 ip-10-0-128-147 kubenswrapper[2558]: E0908 14:24:04.528343    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"288ac4217b90e6208146acd3a6db7b99212afbfa0fbb91a87ba2390f06c3d02d\": container with ID starting with 288ac4217b90e6208146acd3a6db7b99212afbfa0fbb91a87ba2390f06c3d02d not found: ID does not exist" containerID="288ac4217b90e6208146acd3a6db7b99212afbfa0fbb91a87ba2390f06c3d02d"
Sep 08 14:24:04.528431 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.528368    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"288ac4217b90e6208146acd3a6db7b99212afbfa0fbb91a87ba2390f06c3d02d"} err="failed to get container status \"288ac4217b90e6208146acd3a6db7b99212afbfa0fbb91a87ba2390f06c3d02d\": rpc error: code = NotFound desc = could not find container \"288ac4217b90e6208146acd3a6db7b99212afbfa0fbb91a87ba2390f06c3d02d\": container with ID starting with 288ac4217b90e6208146acd3a6db7b99212afbfa0fbb91a87ba2390f06c3d02d not found: ID does not exist"
Sep 08 14:24:04.593236 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.593186    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ba7426f-c99d-4b74-a19e-9895720a156c-cabundle-cert\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:24:04.593236 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.593223    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ba7426f-c99d-4b74-a19e-9895720a156c-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:24:04.593236 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.593234    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prwsp\" (UniqueName: \"kubernetes.io/projected/7ba7426f-c99d-4b74-a19e-9895720a156c-kube-api-access-prwsp\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:24:04.593236 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.593243    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ba7426f-c99d-4b74-a19e-9895720a156c-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:24:04.593631 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.593252    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ba7426f-c99d-4b74-a19e-9895720a156c-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:24:04.593631 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.593261    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ba7426f-c99d-4b74-a19e-9895720a156c-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:24:04.593631 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.593268    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ba7426f-c99d-4b74-a19e-9895720a156c-kserve-provision-location\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:24:04.593631 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.593276    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ba7426f-c99d-4b74-a19e-9895720a156c-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:24:04.831857 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.831821    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"]
Sep 08 14:24:04.837667 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:04.837638    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-5686ffb94f-w4wrk"]
Sep 08 14:24:05.054177 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:05.054141    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ba7426f-c99d-4b74-a19e-9895720a156c" path="/var/lib/kubelet/pods/7ba7426f-c99d-4b74-a19e-9895720a156c/volumes"
Sep 08 14:24:11.207058 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:11.207008    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6" podUID="4eb596d2-d241-4e46-a4eb-b33af3a5011f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 08 14:24:21.207251 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:21.207208    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6" podUID="4eb596d2-d241-4e46-a4eb-b33af3a5011f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 08 14:24:31.207035 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:31.206935    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6" podUID="4eb596d2-d241-4e46-a4eb-b33af3a5011f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 08 14:24:41.206863 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:41.206812    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6" podUID="4eb596d2-d241-4e46-a4eb-b33af3a5011f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 08 14:24:42.117199 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.117157    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"]
Sep 08 14:24:42.117554 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.117541    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ba7426f-c99d-4b74-a19e-9895720a156c" containerName="main"
Sep 08 14:24:42.117624 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.117555    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ba7426f-c99d-4b74-a19e-9895720a156c" containerName="main"
Sep 08 14:24:42.117624 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.117574    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ba7426f-c99d-4b74-a19e-9895720a156c" containerName="storage-initializer"
Sep 08 14:24:42.117624 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.117580    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ba7426f-c99d-4b74-a19e-9895720a156c" containerName="storage-initializer"
Sep 08 14:24:42.117727 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.117647    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ba7426f-c99d-4b74-a19e-9895720a156c" containerName="main"
Sep 08 14:24:42.120915 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.120897    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"
Sep 08 14:24:42.123427 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.123396    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 08 14:24:42.123586 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.123502    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 08 14:24:42.123586 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.123533    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-4b2vf\""
Sep 08 14:24:42.123827 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.123810    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 14:24:42.124259 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.124235    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 08 14:24:42.124259 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.124251    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 08 14:24:42.133509 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.133478    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"]
Sep 08 14:24:42.229977 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.229937    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/623b27e8-4f08-42d6-8da1-de1d3f5cb694-tmp-dir\") pod \"prestop-hook-test-kserve-576469645f-ncpp5\" (UID: \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"
Sep 08 14:24:42.230359 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.229991    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvwzw\" (UniqueName: \"kubernetes.io/projected/623b27e8-4f08-42d6-8da1-de1d3f5cb694-kube-api-access-xvwzw\") pod \"prestop-hook-test-kserve-576469645f-ncpp5\" (UID: \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"
Sep 08 14:24:42.230359 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.230052    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/623b27e8-4f08-42d6-8da1-de1d3f5cb694-kserve-provision-location\") pod \"prestop-hook-test-kserve-576469645f-ncpp5\" (UID: \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"
Sep 08 14:24:42.230359 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.230073    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/623b27e8-4f08-42d6-8da1-de1d3f5cb694-cabundle-cert\") pod \"prestop-hook-test-kserve-576469645f-ncpp5\" (UID: \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"
Sep 08 14:24:42.230359 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.230105    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/623b27e8-4f08-42d6-8da1-de1d3f5cb694-dshm\") pod \"prestop-hook-test-kserve-576469645f-ncpp5\" (UID: \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"
Sep 08 14:24:42.230359 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.230184    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/623b27e8-4f08-42d6-8da1-de1d3f5cb694-model-cache\") pod \"prestop-hook-test-kserve-576469645f-ncpp5\" (UID: \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"
Sep 08 14:24:42.230359 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.230221    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/623b27e8-4f08-42d6-8da1-de1d3f5cb694-home\") pod \"prestop-hook-test-kserve-576469645f-ncpp5\" (UID: \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"
Sep 08 14:24:42.230359 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.230238    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/623b27e8-4f08-42d6-8da1-de1d3f5cb694-tls-certs\") pod \"prestop-hook-test-kserve-576469645f-ncpp5\" (UID: \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"
Sep 08 14:24:42.331306 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.331265    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/623b27e8-4f08-42d6-8da1-de1d3f5cb694-model-cache\") pod \"prestop-hook-test-kserve-576469645f-ncpp5\" (UID: \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"
Sep 08 14:24:42.331449 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.331316    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/623b27e8-4f08-42d6-8da1-de1d3f5cb694-home\") pod \"prestop-hook-test-kserve-576469645f-ncpp5\" (UID: \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"
Sep 08 14:24:42.331449 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.331333    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/623b27e8-4f08-42d6-8da1-de1d3f5cb694-tls-certs\") pod \"prestop-hook-test-kserve-576469645f-ncpp5\" (UID: \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"
Sep 08 14:24:42.331449 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.331375    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/623b27e8-4f08-42d6-8da1-de1d3f5cb694-tmp-dir\") pod \"prestop-hook-test-kserve-576469645f-ncpp5\" (UID: \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"
Sep 08 14:24:42.331449 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.331404    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvwzw\" (UniqueName: \"kubernetes.io/projected/623b27e8-4f08-42d6-8da1-de1d3f5cb694-kube-api-access-xvwzw\") pod \"prestop-hook-test-kserve-576469645f-ncpp5\" (UID: \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"
Sep 08 14:24:42.331718 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.331451    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/623b27e8-4f08-42d6-8da1-de1d3f5cb694-kserve-provision-location\") pod \"prestop-hook-test-kserve-576469645f-ncpp5\" (UID: \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"
Sep 08 14:24:42.331718 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.331478    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/623b27e8-4f08-42d6-8da1-de1d3f5cb694-cabundle-cert\") pod \"prestop-hook-test-kserve-576469645f-ncpp5\" (UID: \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"
Sep 08 14:24:42.331718 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.331569    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/623b27e8-4f08-42d6-8da1-de1d3f5cb694-dshm\") pod \"prestop-hook-test-kserve-576469645f-ncpp5\" (UID: \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"
Sep 08 14:24:42.331875 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.331761    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/623b27e8-4f08-42d6-8da1-de1d3f5cb694-model-cache\") pod \"prestop-hook-test-kserve-576469645f-ncpp5\" (UID: \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"
Sep 08 14:24:42.331875 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.331838    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/623b27e8-4f08-42d6-8da1-de1d3f5cb694-kserve-provision-location\") pod \"prestop-hook-test-kserve-576469645f-ncpp5\" (UID: \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"
Sep 08 14:24:42.332094 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.332070    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/623b27e8-4f08-42d6-8da1-de1d3f5cb694-home\") pod \"prestop-hook-test-kserve-576469645f-ncpp5\" (UID: \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"
Sep 08 14:24:42.332168 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.332120    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/623b27e8-4f08-42d6-8da1-de1d3f5cb694-tmp-dir\") pod \"prestop-hook-test-kserve-576469645f-ncpp5\" (UID: \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"
Sep 08 14:24:42.332486 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.332464    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/623b27e8-4f08-42d6-8da1-de1d3f5cb694-cabundle-cert\") pod \"prestop-hook-test-kserve-576469645f-ncpp5\" (UID: \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"
Sep 08 14:24:42.333780 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.333758    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/623b27e8-4f08-42d6-8da1-de1d3f5cb694-dshm\") pod \"prestop-hook-test-kserve-576469645f-ncpp5\" (UID: \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"
Sep 08 14:24:42.334146 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.334123    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/623b27e8-4f08-42d6-8da1-de1d3f5cb694-tls-certs\") pod \"prestop-hook-test-kserve-576469645f-ncpp5\" (UID: \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"
Sep 08 14:24:42.339402 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.339380    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvwzw\" (UniqueName: \"kubernetes.io/projected/623b27e8-4f08-42d6-8da1-de1d3f5cb694-kube-api-access-xvwzw\") pod \"prestop-hook-test-kserve-576469645f-ncpp5\" (UID: \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"
Sep 08 14:24:42.434498 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.434394    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"
Sep 08 14:24:42.720954 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:42.720796    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"]
Sep 08 14:24:42.723351 ip-10-0-128-147 kubenswrapper[2558]: W0908 14:24:42.723322    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod623b27e8_4f08_42d6_8da1_de1d3f5cb694.slice/crio-d462ba09b92321149dd0f8f2af8fe6339b7783fc9f1c594bd3a7fa02de015ae6 WatchSource:0}: Error finding container d462ba09b92321149dd0f8f2af8fe6339b7783fc9f1c594bd3a7fa02de015ae6: Status 404 returned error can't find the container with id d462ba09b92321149dd0f8f2af8fe6339b7783fc9f1c594bd3a7fa02de015ae6
Sep 08 14:24:43.711319 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:43.711279    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5" event={"ID":"623b27e8-4f08-42d6-8da1-de1d3f5cb694","Type":"ContainerStarted","Data":"e4c648f362e2fcefa8af1459b33d5080eca8814ca96cca1a4db099ebc189a607"}
Sep 08 14:24:43.711846 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:43.711333    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5" event={"ID":"623b27e8-4f08-42d6-8da1-de1d3f5cb694","Type":"ContainerStarted","Data":"d462ba09b92321149dd0f8f2af8fe6339b7783fc9f1c594bd3a7fa02de015ae6"}
Sep 08 14:24:45.723062 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:45.723025    2558 generic.go:356] "Generic (PLEG): container finished" podID="623b27e8-4f08-42d6-8da1-de1d3f5cb694" containerID="e4c648f362e2fcefa8af1459b33d5080eca8814ca96cca1a4db099ebc189a607" exitCode=0
Sep 08 14:24:45.723646 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:45.723098    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5" event={"ID":"623b27e8-4f08-42d6-8da1-de1d3f5cb694","Type":"ContainerDied","Data":"e4c648f362e2fcefa8af1459b33d5080eca8814ca96cca1a4db099ebc189a607"}
Sep 08 14:24:46.731203 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:46.731168    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5" event={"ID":"623b27e8-4f08-42d6-8da1-de1d3f5cb694","Type":"ContainerStarted","Data":"eea96ba722952d75e0eb8cb2275c9779e8a0911aa1847097f84cb13bf662afa8"}
Sep 08 14:24:46.762129 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:46.762053    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5" podStartSLOduration=2.386966636 podStartE2EDuration="4.762035821s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.375069185s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:24:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:24:46.760946724 +0000 UTC m=+4150.212707661" watchObservedRunningTime="2026-09-08 14:24:46.762035821 +0000 UTC m=+4150.213796751"
Sep 08 14:24:47.434764 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:47.434699    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"
Sep 08 14:24:51.206796 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:51.206746    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6" podUID="4eb596d2-d241-4e46-a4eb-b33af3a5011f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 08 14:24:52.435279 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:52.435219    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"
Sep 08 14:24:52.436863 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:24:52.436832    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5" podUID="623b27e8-4f08-42d6-8da1-de1d3f5cb694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 08 14:25:01.207199 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:01.207140    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6" podUID="4eb596d2-d241-4e46-a4eb-b33af3a5011f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 08 14:25:02.435552 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:02.435511    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5" podUID="623b27e8-4f08-42d6-8da1-de1d3f5cb694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 08 14:25:11.206611 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:11.206544    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6" podUID="4eb596d2-d241-4e46-a4eb-b33af3a5011f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 08 14:25:12.435503 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:12.435455    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5" podUID="623b27e8-4f08-42d6-8da1-de1d3f5cb694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 08 14:25:21.207029 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:21.206980    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6" podUID="4eb596d2-d241-4e46-a4eb-b33af3a5011f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 08 14:25:22.434911 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:22.434846    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5" podUID="623b27e8-4f08-42d6-8da1-de1d3f5cb694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 08 14:25:31.217321 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:31.217283    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"
Sep 08 14:25:31.225790 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:31.225762    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"
Sep 08 14:25:32.435717 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:32.435668    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5" podUID="623b27e8-4f08-42d6-8da1-de1d3f5cb694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 08 14:25:40.370243 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:40.370190    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"]
Sep 08 14:25:42.435056 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:42.435004    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5" podUID="623b27e8-4f08-42d6-8da1-de1d3f5cb694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 08 14:25:52.435043 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:52.434998    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5" podUID="623b27e8-4f08-42d6-8da1-de1d3f5cb694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 08 14:25:55.393737 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:55.393696    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6" podUID="4eb596d2-d241-4e46-a4eb-b33af3a5011f" containerName="main" containerID="cri-o://6661db98cd82856e80c8c1acdc33aeb204b3f5790697c1d61c8b74ac067fc3e9" gracePeriod=45
Sep 08 14:25:56.118996 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:56.118964    2558 generic.go:356] "Generic (PLEG): container finished" podID="4eb596d2-d241-4e46-a4eb-b33af3a5011f" containerID="6661db98cd82856e80c8c1acdc33aeb204b3f5790697c1d61c8b74ac067fc3e9" exitCode=0
Sep 08 14:25:56.119150 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:56.119036    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6" event={"ID":"4eb596d2-d241-4e46-a4eb-b33af3a5011f","Type":"ContainerDied","Data":"6661db98cd82856e80c8c1acdc33aeb204b3f5790697c1d61c8b74ac067fc3e9"}
Sep 08 14:25:56.144000 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:56.143975    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"
Sep 08 14:25:56.240976 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:56.240883    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4eb596d2-d241-4e46-a4eb-b33af3a5011f-kserve-provision-location\") pod \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\" (UID: \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\") "
Sep 08 14:25:56.240976 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:56.240927    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eb596d2-d241-4e46-a4eb-b33af3a5011f-dshm\") pod \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\" (UID: \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\") "
Sep 08 14:25:56.240976 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:56.240952    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eb596d2-d241-4e46-a4eb-b33af3a5011f-home\") pod \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\" (UID: \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\") "
Sep 08 14:25:56.240976 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:56.240978    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eb596d2-d241-4e46-a4eb-b33af3a5011f-model-cache\") pod \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\" (UID: \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\") "
Sep 08 14:25:56.241315 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:56.241024    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eb596d2-d241-4e46-a4eb-b33af3a5011f-tmp-dir\") pod \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\" (UID: \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\") "
Sep 08 14:25:56.241315 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:56.241041    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4eb596d2-d241-4e46-a4eb-b33af3a5011f-cabundle-cert\") pod \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\" (UID: \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\") "
Sep 08 14:25:56.241315 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:56.241100    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ntgb\" (UniqueName: \"kubernetes.io/projected/4eb596d2-d241-4e46-a4eb-b33af3a5011f-kube-api-access-4ntgb\") pod \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\" (UID: \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\") "
Sep 08 14:25:56.241315 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:56.241123    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eb596d2-d241-4e46-a4eb-b33af3a5011f-tls-certs\") pod \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\" (UID: \"4eb596d2-d241-4e46-a4eb-b33af3a5011f\") "
Sep 08 14:25:56.241315 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:56.241293    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb596d2-d241-4e46-a4eb-b33af3a5011f-model-cache" (OuterVolumeSpecName: "model-cache") pod "4eb596d2-d241-4e46-a4eb-b33af3a5011f" (UID: "4eb596d2-d241-4e46-a4eb-b33af3a5011f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:25:56.241634 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:56.241418    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4eb596d2-d241-4e46-a4eb-b33af3a5011f-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:25:56.241948 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:56.241740    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb596d2-d241-4e46-a4eb-b33af3a5011f-home" (OuterVolumeSpecName: "home") pod "4eb596d2-d241-4e46-a4eb-b33af3a5011f" (UID: "4eb596d2-d241-4e46-a4eb-b33af3a5011f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:25:56.241948 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:56.241899    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb596d2-d241-4e46-a4eb-b33af3a5011f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4eb596d2-d241-4e46-a4eb-b33af3a5011f" (UID: "4eb596d2-d241-4e46-a4eb-b33af3a5011f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:25:56.243767 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:56.243738    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb596d2-d241-4e46-a4eb-b33af3a5011f-dshm" (OuterVolumeSpecName: "dshm") pod "4eb596d2-d241-4e46-a4eb-b33af3a5011f" (UID: "4eb596d2-d241-4e46-a4eb-b33af3a5011f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:25:56.244000 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:56.243980    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eb596d2-d241-4e46-a4eb-b33af3a5011f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4eb596d2-d241-4e46-a4eb-b33af3a5011f" (UID: "4eb596d2-d241-4e46-a4eb-b33af3a5011f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:25:56.244067 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:56.243995    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb596d2-d241-4e46-a4eb-b33af3a5011f-kube-api-access-4ntgb" (OuterVolumeSpecName: "kube-api-access-4ntgb") pod "4eb596d2-d241-4e46-a4eb-b33af3a5011f" (UID: "4eb596d2-d241-4e46-a4eb-b33af3a5011f"). InnerVolumeSpecName "kube-api-access-4ntgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:25:56.253417 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:56.253379    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb596d2-d241-4e46-a4eb-b33af3a5011f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4eb596d2-d241-4e46-a4eb-b33af3a5011f" (UID: "4eb596d2-d241-4e46-a4eb-b33af3a5011f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:25:56.266039 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:56.266003    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb596d2-d241-4e46-a4eb-b33af3a5011f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4eb596d2-d241-4e46-a4eb-b33af3a5011f" (UID: "4eb596d2-d241-4e46-a4eb-b33af3a5011f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:25:56.342610 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:56.342554    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4eb596d2-d241-4e46-a4eb-b33af3a5011f-kserve-provision-location\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:25:56.342610 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:56.342588    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4eb596d2-d241-4e46-a4eb-b33af3a5011f-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:25:56.342610 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:56.342616    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4eb596d2-d241-4e46-a4eb-b33af3a5011f-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:25:56.342819 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:56.342624    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4eb596d2-d241-4e46-a4eb-b33af3a5011f-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:25:56.342819 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:56.342633    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4eb596d2-d241-4e46-a4eb-b33af3a5011f-cabundle-cert\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:25:56.342819 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:56.342641    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ntgb\" (UniqueName: \"kubernetes.io/projected/4eb596d2-d241-4e46-a4eb-b33af3a5011f-kube-api-access-4ntgb\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:25:56.342819 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:56.342648    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eb596d2-d241-4e46-a4eb-b33af3a5011f-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:25:57.126103 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:57.126070    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6" event={"ID":"4eb596d2-d241-4e46-a4eb-b33af3a5011f","Type":"ContainerDied","Data":"6a86e66c4b49139beb54acf0f463e5142854fb61e5ec33112dc524cad74ee20c"}
Sep 08 14:25:57.126103 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:57.126092    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"
Sep 08 14:25:57.126682 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:57.126132    2558 scope.go:117] "RemoveContainer" containerID="6661db98cd82856e80c8c1acdc33aeb204b3f5790697c1d61c8b74ac067fc3e9"
Sep 08 14:25:57.137136 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:57.137112    2558 scope.go:117] "RemoveContainer" containerID="f51ff2ba1e757e73f89c0f9fbbcb9311d886529f97b9b8fe5d027ebc54e8bdb2"
Sep 08 14:25:57.143779 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:57.143744    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"]
Sep 08 14:25:57.149965 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:57.149937    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-694c44db4b-f6qr6"]
Sep 08 14:25:59.054505 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:25:59.054464    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4eb596d2-d241-4e46-a4eb-b33af3a5011f" path="/var/lib/kubelet/pods/4eb596d2-d241-4e46-a4eb-b33af3a5011f/volumes"
Sep 08 14:26:02.435394 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:26:02.435353    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5" podUID="623b27e8-4f08-42d6-8da1-de1d3f5cb694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 08 14:26:12.435269 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:26:12.435220    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5" podUID="623b27e8-4f08-42d6-8da1-de1d3f5cb694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 08 14:26:22.434875 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:26:22.434829    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5" podUID="623b27e8-4f08-42d6-8da1-de1d3f5cb694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 08 14:26:32.435229 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:26:32.435182    2558 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5" podUID="623b27e8-4f08-42d6-8da1-de1d3f5cb694" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Sep 08 14:26:42.445388 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:26:42.445306    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"
Sep 08 14:26:42.453308 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:26:42.453282    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"
Sep 08 14:26:44.459325 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:26:44.459291    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"]
Sep 08 14:26:59.482095 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:26:59.482045    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5" podUID="623b27e8-4f08-42d6-8da1-de1d3f5cb694" containerName="main" containerID="cri-o://eea96ba722952d75e0eb8cb2275c9779e8a0911aa1847097f84cb13bf662afa8" gracePeriod=45
Sep 08 14:27:00.148567 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.148537    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"
Sep 08 14:27:00.317693 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.317600    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/623b27e8-4f08-42d6-8da1-de1d3f5cb694-tls-certs\") pod \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\" (UID: \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\") "
Sep 08 14:27:00.317693 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.317640    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/623b27e8-4f08-42d6-8da1-de1d3f5cb694-model-cache\") pod \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\" (UID: \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\") "
Sep 08 14:27:00.317693 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.317661    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvwzw\" (UniqueName: \"kubernetes.io/projected/623b27e8-4f08-42d6-8da1-de1d3f5cb694-kube-api-access-xvwzw\") pod \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\" (UID: \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\") "
Sep 08 14:27:00.317693 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.317691    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/623b27e8-4f08-42d6-8da1-de1d3f5cb694-dshm\") pod \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\" (UID: \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\") "
Sep 08 14:27:00.318041 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.317721    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/623b27e8-4f08-42d6-8da1-de1d3f5cb694-kserve-provision-location\") pod \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\" (UID: \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\") "
Sep 08 14:27:00.318041 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.317770    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/623b27e8-4f08-42d6-8da1-de1d3f5cb694-cabundle-cert\") pod \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\" (UID: \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\") "
Sep 08 14:27:00.318041 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.317797    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/623b27e8-4f08-42d6-8da1-de1d3f5cb694-home\") pod \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\" (UID: \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\") "
Sep 08 14:27:00.318041 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.317846    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/623b27e8-4f08-42d6-8da1-de1d3f5cb694-tmp-dir\") pod \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\" (UID: \"623b27e8-4f08-42d6-8da1-de1d3f5cb694\") "
Sep 08 14:27:00.318041 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.317976    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/623b27e8-4f08-42d6-8da1-de1d3f5cb694-model-cache" (OuterVolumeSpecName: "model-cache") pod "623b27e8-4f08-42d6-8da1-de1d3f5cb694" (UID: "623b27e8-4f08-42d6-8da1-de1d3f5cb694"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:27:00.318285 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.318149    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/623b27e8-4f08-42d6-8da1-de1d3f5cb694-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:27:00.318622 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.318531    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/623b27e8-4f08-42d6-8da1-de1d3f5cb694-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "623b27e8-4f08-42d6-8da1-de1d3f5cb694" (UID: "623b27e8-4f08-42d6-8da1-de1d3f5cb694"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:27:00.318742 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.318620    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/623b27e8-4f08-42d6-8da1-de1d3f5cb694-home" (OuterVolumeSpecName: "home") pod "623b27e8-4f08-42d6-8da1-de1d3f5cb694" (UID: "623b27e8-4f08-42d6-8da1-de1d3f5cb694"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:27:00.320316 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.320282    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/623b27e8-4f08-42d6-8da1-de1d3f5cb694-dshm" (OuterVolumeSpecName: "dshm") pod "623b27e8-4f08-42d6-8da1-de1d3f5cb694" (UID: "623b27e8-4f08-42d6-8da1-de1d3f5cb694"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:27:00.320429 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.320336    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/623b27e8-4f08-42d6-8da1-de1d3f5cb694-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "623b27e8-4f08-42d6-8da1-de1d3f5cb694" (UID: "623b27e8-4f08-42d6-8da1-de1d3f5cb694"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:27:00.320429 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.320355    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/623b27e8-4f08-42d6-8da1-de1d3f5cb694-kube-api-access-xvwzw" (OuterVolumeSpecName: "kube-api-access-xvwzw") pod "623b27e8-4f08-42d6-8da1-de1d3f5cb694" (UID: "623b27e8-4f08-42d6-8da1-de1d3f5cb694"). InnerVolumeSpecName "kube-api-access-xvwzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:27:00.332137 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.332112    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/623b27e8-4f08-42d6-8da1-de1d3f5cb694-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "623b27e8-4f08-42d6-8da1-de1d3f5cb694" (UID: "623b27e8-4f08-42d6-8da1-de1d3f5cb694"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:27:00.341937 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.341908    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/623b27e8-4f08-42d6-8da1-de1d3f5cb694-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "623b27e8-4f08-42d6-8da1-de1d3f5cb694" (UID: "623b27e8-4f08-42d6-8da1-de1d3f5cb694"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:27:00.419173 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.419135    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/623b27e8-4f08-42d6-8da1-de1d3f5cb694-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:27:00.419173 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.419168    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvwzw\" (UniqueName: \"kubernetes.io/projected/623b27e8-4f08-42d6-8da1-de1d3f5cb694-kube-api-access-xvwzw\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:27:00.419173 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.419180    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/623b27e8-4f08-42d6-8da1-de1d3f5cb694-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:27:00.419409 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.419192    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/623b27e8-4f08-42d6-8da1-de1d3f5cb694-kserve-provision-location\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:27:00.419409 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.419205    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/623b27e8-4f08-42d6-8da1-de1d3f5cb694-cabundle-cert\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:27:00.419409 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.419215    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/623b27e8-4f08-42d6-8da1-de1d3f5cb694-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:27:00.419409 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.419225    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/623b27e8-4f08-42d6-8da1-de1d3f5cb694-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:27:00.446988 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.446960    2558 generic.go:356] "Generic (PLEG): container finished" podID="623b27e8-4f08-42d6-8da1-de1d3f5cb694" containerID="eea96ba722952d75e0eb8cb2275c9779e8a0911aa1847097f84cb13bf662afa8" exitCode=0
Sep 08 14:27:00.447126 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.447010    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5" event={"ID":"623b27e8-4f08-42d6-8da1-de1d3f5cb694","Type":"ContainerDied","Data":"eea96ba722952d75e0eb8cb2275c9779e8a0911aa1847097f84cb13bf662afa8"}
Sep 08 14:27:00.447126 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.447018    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"
Sep 08 14:27:00.447126 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.447053    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5" event={"ID":"623b27e8-4f08-42d6-8da1-de1d3f5cb694","Type":"ContainerDied","Data":"d462ba09b92321149dd0f8f2af8fe6339b7783fc9f1c594bd3a7fa02de015ae6"}
Sep 08 14:27:00.447126 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.447074    2558 scope.go:117] "RemoveContainer" containerID="eea96ba722952d75e0eb8cb2275c9779e8a0911aa1847097f84cb13bf662afa8"
Sep 08 14:27:00.458381 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.458153    2558 scope.go:117] "RemoveContainer" containerID="e4c648f362e2fcefa8af1459b33d5080eca8814ca96cca1a4db099ebc189a607"
Sep 08 14:27:00.466280 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.466263    2558 scope.go:117] "RemoveContainer" containerID="eea96ba722952d75e0eb8cb2275c9779e8a0911aa1847097f84cb13bf662afa8"
Sep 08 14:27:00.466533 ip-10-0-128-147 kubenswrapper[2558]: E0908 14:27:00.466514    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eea96ba722952d75e0eb8cb2275c9779e8a0911aa1847097f84cb13bf662afa8\": container with ID starting with eea96ba722952d75e0eb8cb2275c9779e8a0911aa1847097f84cb13bf662afa8 not found: ID does not exist" containerID="eea96ba722952d75e0eb8cb2275c9779e8a0911aa1847097f84cb13bf662afa8"
Sep 08 14:27:00.466618 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.466543    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eea96ba722952d75e0eb8cb2275c9779e8a0911aa1847097f84cb13bf662afa8"} err="failed to get container status \"eea96ba722952d75e0eb8cb2275c9779e8a0911aa1847097f84cb13bf662afa8\": rpc error: code = NotFound desc = could not find container \"eea96ba722952d75e0eb8cb2275c9779e8a0911aa1847097f84cb13bf662afa8\": container with ID starting with eea96ba722952d75e0eb8cb2275c9779e8a0911aa1847097f84cb13bf662afa8 not found: ID does not exist"
Sep 08 14:27:00.466618 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.466559    2558 scope.go:117] "RemoveContainer" containerID="e4c648f362e2fcefa8af1459b33d5080eca8814ca96cca1a4db099ebc189a607"
Sep 08 14:27:00.466863 ip-10-0-128-147 kubenswrapper[2558]: E0908 14:27:00.466845    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4c648f362e2fcefa8af1459b33d5080eca8814ca96cca1a4db099ebc189a607\": container with ID starting with e4c648f362e2fcefa8af1459b33d5080eca8814ca96cca1a4db099ebc189a607 not found: ID does not exist" containerID="e4c648f362e2fcefa8af1459b33d5080eca8814ca96cca1a4db099ebc189a607"
Sep 08 14:27:00.466904 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.466868    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4c648f362e2fcefa8af1459b33d5080eca8814ca96cca1a4db099ebc189a607"} err="failed to get container status \"e4c648f362e2fcefa8af1459b33d5080eca8814ca96cca1a4db099ebc189a607\": rpc error: code = NotFound desc = could not find container \"e4c648f362e2fcefa8af1459b33d5080eca8814ca96cca1a4db099ebc189a607\": container with ID starting with e4c648f362e2fcefa8af1459b33d5080eca8814ca96cca1a4db099ebc189a607 not found: ID does not exist"
Sep 08 14:27:00.470640 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.470618    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"]
Sep 08 14:27:00.473921 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:00.473900    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-576469645f-ncpp5"]
Sep 08 14:27:01.053580 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:01.053542    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="623b27e8-4f08-42d6-8da1-de1d3f5cb694" path="/var/lib/kubelet/pods/623b27e8-4f08-42d6-8da1-de1d3f5cb694/volumes"
Sep 08 14:27:24.936115 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:24.936075    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w"]
Sep 08 14:27:24.936557 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:24.936444    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="623b27e8-4f08-42d6-8da1-de1d3f5cb694" containerName="storage-initializer"
Sep 08 14:27:24.936557 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:24.936455    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="623b27e8-4f08-42d6-8da1-de1d3f5cb694" containerName="storage-initializer"
Sep 08 14:27:24.936557 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:24.936463    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="623b27e8-4f08-42d6-8da1-de1d3f5cb694" containerName="main"
Sep 08 14:27:24.936557 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:24.936470    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="623b27e8-4f08-42d6-8da1-de1d3f5cb694" containerName="main"
Sep 08 14:27:24.936557 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:24.936478    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eb596d2-d241-4e46-a4eb-b33af3a5011f" containerName="main"
Sep 08 14:27:24.936557 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:24.936486    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eb596d2-d241-4e46-a4eb-b33af3a5011f" containerName="main"
Sep 08 14:27:24.936557 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:24.936492    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eb596d2-d241-4e46-a4eb-b33af3a5011f" containerName="storage-initializer"
Sep 08 14:27:24.936557 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:24.936498    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eb596d2-d241-4e46-a4eb-b33af3a5011f" containerName="storage-initializer"
Sep 08 14:27:24.936557 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:24.936552    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eb596d2-d241-4e46-a4eb-b33af3a5011f" containerName="main"
Sep 08 14:27:24.936557 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:24.936561    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="623b27e8-4f08-42d6-8da1-de1d3f5cb694" containerName="main"
Sep 08 14:27:24.939490 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:24.939471    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w"
Sep 08 14:27:24.941739 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:24.941717    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 08 14:27:24.942413 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:24.942388    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-flfhf\""
Sep 08 14:27:24.942527 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:24.942474    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 08 14:27:24.942527 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:24.942504    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 08 14:27:24.948123 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:24.948097    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w"]
Sep 08 14:27:25.035867 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:25.035829    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a52c73-052d-4ab6-abce-0070743994f6-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-b6v9w\" (UID: \"b6a52c73-052d-4ab6-abce-0070743994f6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w"
Sep 08 14:27:25.036062 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:25.035884    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a52c73-052d-4ab6-abce-0070743994f6-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-b6v9w\" (UID: \"b6a52c73-052d-4ab6-abce-0070743994f6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w"
Sep 08 14:27:25.036062 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:25.035911    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grlc9\" (UniqueName: \"kubernetes.io/projected/b6a52c73-052d-4ab6-abce-0070743994f6-kube-api-access-grlc9\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-b6v9w\" (UID: \"b6a52c73-052d-4ab6-abce-0070743994f6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w"
Sep 08 14:27:25.036062 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:25.035931    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a52c73-052d-4ab6-abce-0070743994f6-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-b6v9w\" (UID: \"b6a52c73-052d-4ab6-abce-0070743994f6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w"
Sep 08 14:27:25.036062 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:25.035953    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a52c73-052d-4ab6-abce-0070743994f6-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-b6v9w\" (UID: \"b6a52c73-052d-4ab6-abce-0070743994f6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w"
Sep 08 14:27:25.036062 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:25.035974    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a52c73-052d-4ab6-abce-0070743994f6-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-b6v9w\" (UID: \"b6a52c73-052d-4ab6-abce-0070743994f6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w"
Sep 08 14:27:25.137272 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:25.137229    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a52c73-052d-4ab6-abce-0070743994f6-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-b6v9w\" (UID: \"b6a52c73-052d-4ab6-abce-0070743994f6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w"
Sep 08 14:27:25.137482 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:25.137288    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a52c73-052d-4ab6-abce-0070743994f6-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-b6v9w\" (UID: \"b6a52c73-052d-4ab6-abce-0070743994f6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w"
Sep 08 14:27:25.137482 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:25.137309    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grlc9\" (UniqueName: \"kubernetes.io/projected/b6a52c73-052d-4ab6-abce-0070743994f6-kube-api-access-grlc9\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-b6v9w\" (UID: \"b6a52c73-052d-4ab6-abce-0070743994f6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w"
Sep 08 14:27:25.137482 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:25.137335    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a52c73-052d-4ab6-abce-0070743994f6-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-b6v9w\" (UID: \"b6a52c73-052d-4ab6-abce-0070743994f6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w"
Sep 08 14:27:25.137482 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:25.137358    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a52c73-052d-4ab6-abce-0070743994f6-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-b6v9w\" (UID: \"b6a52c73-052d-4ab6-abce-0070743994f6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w"
Sep 08 14:27:25.137482 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:25.137382    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a52c73-052d-4ab6-abce-0070743994f6-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-b6v9w\" (UID: \"b6a52c73-052d-4ab6-abce-0070743994f6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w"
Sep 08 14:27:25.137796 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:25.137699    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a52c73-052d-4ab6-abce-0070743994f6-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-b6v9w\" (UID: \"b6a52c73-052d-4ab6-abce-0070743994f6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w"
Sep 08 14:27:25.137870 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:25.137851    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a52c73-052d-4ab6-abce-0070743994f6-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-b6v9w\" (UID: \"b6a52c73-052d-4ab6-abce-0070743994f6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w"
Sep 08 14:27:25.137907 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:25.137887    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a52c73-052d-4ab6-abce-0070743994f6-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-b6v9w\" (UID: \"b6a52c73-052d-4ab6-abce-0070743994f6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w"
Sep 08 14:27:25.139681 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:25.139660    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a52c73-052d-4ab6-abce-0070743994f6-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-b6v9w\" (UID: \"b6a52c73-052d-4ab6-abce-0070743994f6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w"
Sep 08 14:27:25.140088 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:25.140066    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a52c73-052d-4ab6-abce-0070743994f6-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-b6v9w\" (UID: \"b6a52c73-052d-4ab6-abce-0070743994f6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w"
Sep 08 14:27:25.144997 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:25.144972    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grlc9\" (UniqueName: \"kubernetes.io/projected/b6a52c73-052d-4ab6-abce-0070743994f6-kube-api-access-grlc9\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-b6v9w\" (UID: \"b6a52c73-052d-4ab6-abce-0070743994f6\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w"
Sep 08 14:27:25.250774 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:25.250676    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w"
Sep 08 14:27:25.522988 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:25.522906    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w"]
Sep 08 14:27:25.526282 ip-10-0-128-147 kubenswrapper[2558]: W0908 14:27:25.526252    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a52c73_052d_4ab6_abce_0070743994f6.slice/crio-dffa6973c5232f684e08b621fa418acc6f6fca00f7d4492a4520a72b692bdf8e WatchSource:0}: Error finding container dffa6973c5232f684e08b621fa418acc6f6fca00f7d4492a4520a72b692bdf8e: Status 404 returned error can't find the container with id dffa6973c5232f684e08b621fa418acc6f6fca00f7d4492a4520a72b692bdf8e
Sep 08 14:27:25.569788 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:25.569760    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w" event={"ID":"b6a52c73-052d-4ab6-abce-0070743994f6","Type":"ContainerStarted","Data":"dffa6973c5232f684e08b621fa418acc6f6fca00f7d4492a4520a72b692bdf8e"}
Sep 08 14:27:26.576733 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:26.576694    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w" event={"ID":"b6a52c73-052d-4ab6-abce-0070743994f6","Type":"ContainerStarted","Data":"e77990344ef414b0b42281ecf185de9048751162f1e62ea033796ed72365dbf7"}
Sep 08 14:27:26.601406 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:26.601333    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w" podStartSLOduration=2.601309648 podStartE2EDuration="2.601309648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:27:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:27:26.599446334 +0000 UTC m=+4310.051207266" watchObservedRunningTime="2026-09-08 14:27:26.601309648 +0000 UTC m=+4310.053070606"
Sep 08 14:27:27.250862 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:27.250823    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w"
Sep 08 14:27:30.573361 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:30.573267    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w"]
Sep 08 14:27:30.583113 ip-10-0-128-147 kubenswrapper[2558]: E0908 14:27:30.583072    2558 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 08 14:27:30.583113 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 14:27:30.583113 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 14:27:30.583113 ip-10-0-128-147 kubenswrapper[2558]:  > containerID="e77990344ef414b0b42281ecf185de9048751162f1e62ea033796ed72365dbf7" cmd=["/bin/sleep","15"]
Sep 08 14:27:30.583405 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:30.583132    2558 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w" message="" err=<
Sep 08 14:27:30.583405 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 14:27:30.583405 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 14:27:30.583405 ip-10-0-128-147 kubenswrapper[2558]:  >
Sep 08 14:27:30.583405 ip-10-0-128-147 kubenswrapper[2558]: E0908 14:27:30.583145    2558 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 08 14:27:30.583405 ip-10-0-128-147 kubenswrapper[2558]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 08 14:27:30.583405 ip-10-0-128-147 kubenswrapper[2558]:         , stdout: , stderr: , exit code -1
Sep 08 14:27:30.583405 ip-10-0-128-147 kubenswrapper[2558]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w" podUID="b6a52c73-052d-4ab6-abce-0070743994f6" containerName="main" containerID="cri-o://e77990344ef414b0b42281ecf185de9048751162f1e62ea033796ed72365dbf7"
Sep 08 14:27:30.583405 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:30.583179    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w" podUID="b6a52c73-052d-4ab6-abce-0070743994f6" containerName="main" containerID="cri-o://e77990344ef414b0b42281ecf185de9048751162f1e62ea033796ed72365dbf7" gracePeriod=60
Sep 08 14:27:30.940838 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:30.940807    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w"
Sep 08 14:27:31.089801 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:31.089771    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a52c73-052d-4ab6-abce-0070743994f6-dshm\") pod \"b6a52c73-052d-4ab6-abce-0070743994f6\" (UID: \"b6a52c73-052d-4ab6-abce-0070743994f6\") "
Sep 08 14:27:31.089994 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:31.089823    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a52c73-052d-4ab6-abce-0070743994f6-tls-certs\") pod \"b6a52c73-052d-4ab6-abce-0070743994f6\" (UID: \"b6a52c73-052d-4ab6-abce-0070743994f6\") "
Sep 08 14:27:31.089994 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:31.089860    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a52c73-052d-4ab6-abce-0070743994f6-model-cache\") pod \"b6a52c73-052d-4ab6-abce-0070743994f6\" (UID: \"b6a52c73-052d-4ab6-abce-0070743994f6\") "
Sep 08 14:27:31.089994 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:31.089888    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grlc9\" (UniqueName: \"kubernetes.io/projected/b6a52c73-052d-4ab6-abce-0070743994f6-kube-api-access-grlc9\") pod \"b6a52c73-052d-4ab6-abce-0070743994f6\" (UID: \"b6a52c73-052d-4ab6-abce-0070743994f6\") "
Sep 08 14:27:31.089994 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:31.089911    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a52c73-052d-4ab6-abce-0070743994f6-home\") pod \"b6a52c73-052d-4ab6-abce-0070743994f6\" (UID: \"b6a52c73-052d-4ab6-abce-0070743994f6\") "
Sep 08 14:27:31.089994 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:31.089936    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a52c73-052d-4ab6-abce-0070743994f6-tmp-dir\") pod \"b6a52c73-052d-4ab6-abce-0070743994f6\" (UID: \"b6a52c73-052d-4ab6-abce-0070743994f6\") "
Sep 08 14:27:31.090268 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:31.090129    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a52c73-052d-4ab6-abce-0070743994f6-model-cache" (OuterVolumeSpecName: "model-cache") pod "b6a52c73-052d-4ab6-abce-0070743994f6" (UID: "b6a52c73-052d-4ab6-abce-0070743994f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:27:31.090268 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:31.090224    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a52c73-052d-4ab6-abce-0070743994f6-home" (OuterVolumeSpecName: "home") pod "b6a52c73-052d-4ab6-abce-0070743994f6" (UID: "b6a52c73-052d-4ab6-abce-0070743994f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:27:31.090389 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:31.090348    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a52c73-052d-4ab6-abce-0070743994f6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b6a52c73-052d-4ab6-abce-0070743994f6" (UID: "b6a52c73-052d-4ab6-abce-0070743994f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:27:31.092219 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:31.092188    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a52c73-052d-4ab6-abce-0070743994f6-dshm" (OuterVolumeSpecName: "dshm") pod "b6a52c73-052d-4ab6-abce-0070743994f6" (UID: "b6a52c73-052d-4ab6-abce-0070743994f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:27:31.092219 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:31.092201    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6a52c73-052d-4ab6-abce-0070743994f6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b6a52c73-052d-4ab6-abce-0070743994f6" (UID: "b6a52c73-052d-4ab6-abce-0070743994f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:27:31.092219 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:31.092202    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a52c73-052d-4ab6-abce-0070743994f6-kube-api-access-grlc9" (OuterVolumeSpecName: "kube-api-access-grlc9") pod "b6a52c73-052d-4ab6-abce-0070743994f6" (UID: "b6a52c73-052d-4ab6-abce-0070743994f6"). InnerVolumeSpecName "kube-api-access-grlc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:27:31.191046 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:31.191004    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a52c73-052d-4ab6-abce-0070743994f6-model-cache\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:27:31.191046 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:31.191040    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grlc9\" (UniqueName: \"kubernetes.io/projected/b6a52c73-052d-4ab6-abce-0070743994f6-kube-api-access-grlc9\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:27:31.191046 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:31.191051    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a52c73-052d-4ab6-abce-0070743994f6-home\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:27:31.191046 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:31.191061    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a52c73-052d-4ab6-abce-0070743994f6-tmp-dir\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:27:31.191341 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:31.191069    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a52c73-052d-4ab6-abce-0070743994f6-dshm\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:27:31.191341 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:31.191077    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a52c73-052d-4ab6-abce-0070743994f6-tls-certs\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:27:31.602639 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:31.602522    2558 generic.go:356] "Generic (PLEG): container finished" podID="b6a52c73-052d-4ab6-abce-0070743994f6" containerID="e77990344ef414b0b42281ecf185de9048751162f1e62ea033796ed72365dbf7" exitCode=0
Sep 08 14:27:31.602639 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:31.602568    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w"
Sep 08 14:27:31.602639 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:31.602629    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w" event={"ID":"b6a52c73-052d-4ab6-abce-0070743994f6","Type":"ContainerDied","Data":"e77990344ef414b0b42281ecf185de9048751162f1e62ea033796ed72365dbf7"}
Sep 08 14:27:31.603255 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:31.602689    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w" event={"ID":"b6a52c73-052d-4ab6-abce-0070743994f6","Type":"ContainerDied","Data":"dffa6973c5232f684e08b621fa418acc6f6fca00f7d4492a4520a72b692bdf8e"}
Sep 08 14:27:31.603255 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:31.602722    2558 scope.go:117] "RemoveContainer" containerID="e77990344ef414b0b42281ecf185de9048751162f1e62ea033796ed72365dbf7"
Sep 08 14:27:31.612113 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:31.612092    2558 scope.go:117] "RemoveContainer" containerID="e77990344ef414b0b42281ecf185de9048751162f1e62ea033796ed72365dbf7"
Sep 08 14:27:31.612382 ip-10-0-128-147 kubenswrapper[2558]: E0908 14:27:31.612362    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e77990344ef414b0b42281ecf185de9048751162f1e62ea033796ed72365dbf7\": container with ID starting with e77990344ef414b0b42281ecf185de9048751162f1e62ea033796ed72365dbf7 not found: ID does not exist" containerID="e77990344ef414b0b42281ecf185de9048751162f1e62ea033796ed72365dbf7"
Sep 08 14:27:31.612434 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:31.612390    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e77990344ef414b0b42281ecf185de9048751162f1e62ea033796ed72365dbf7"} err="failed to get container status \"e77990344ef414b0b42281ecf185de9048751162f1e62ea033796ed72365dbf7\": rpc error: code = NotFound desc = could not find container \"e77990344ef414b0b42281ecf185de9048751162f1e62ea033796ed72365dbf7\": container with ID starting with e77990344ef414b0b42281ecf185de9048751162f1e62ea033796ed72365dbf7 not found: ID does not exist"
Sep 08 14:27:31.628020 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:31.627978    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w"]
Sep 08 14:27:31.633489 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:31.633466    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-b6v9w"]
Sep 08 14:27:33.053739 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:27:33.053702    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6a52c73-052d-4ab6-abce-0070743994f6" path="/var/lib/kubelet/pods/b6a52c73-052d-4ab6-abce-0070743994f6/volumes"
Sep 08 14:28:20.229604 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:20.229564    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tg9wr/must-gather-6ss7s"]
Sep 08 14:28:20.230074 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:20.229939    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6a52c73-052d-4ab6-abce-0070743994f6" containerName="main"
Sep 08 14:28:20.230074 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:20.229951    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6a52c73-052d-4ab6-abce-0070743994f6" containerName="main"
Sep 08 14:28:20.230074 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:20.230016    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6a52c73-052d-4ab6-abce-0070743994f6" containerName="main"
Sep 08 14:28:20.233192 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:20.233174    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tg9wr/must-gather-6ss7s"
Sep 08 14:28:20.236243 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:20.236205    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tg9wr\"/\"default-dockercfg-hdqst\""
Sep 08 14:28:20.236409 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:20.236260    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tg9wr\"/\"kube-root-ca.crt\""
Sep 08 14:28:20.236409 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:20.236370    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tg9wr\"/\"openshift-service-ca.crt\""
Sep 08 14:28:20.246065 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:20.246038    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tg9wr/must-gather-6ss7s"]
Sep 08 14:28:20.300691 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:20.300649    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zz6h\" (UniqueName: \"kubernetes.io/projected/5762e105-bdd0-49fd-a477-4dfc9b5ae71d-kube-api-access-6zz6h\") pod \"must-gather-6ss7s\" (UID: \"5762e105-bdd0-49fd-a477-4dfc9b5ae71d\") " pod="openshift-must-gather-tg9wr/must-gather-6ss7s"
Sep 08 14:28:20.300878 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:20.300712    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5762e105-bdd0-49fd-a477-4dfc9b5ae71d-must-gather-output\") pod \"must-gather-6ss7s\" (UID: \"5762e105-bdd0-49fd-a477-4dfc9b5ae71d\") " pod="openshift-must-gather-tg9wr/must-gather-6ss7s"
Sep 08 14:28:20.401922 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:20.401880    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5762e105-bdd0-49fd-a477-4dfc9b5ae71d-must-gather-output\") pod \"must-gather-6ss7s\" (UID: \"5762e105-bdd0-49fd-a477-4dfc9b5ae71d\") " pod="openshift-must-gather-tg9wr/must-gather-6ss7s"
Sep 08 14:28:20.402092 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:20.401961    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zz6h\" (UniqueName: \"kubernetes.io/projected/5762e105-bdd0-49fd-a477-4dfc9b5ae71d-kube-api-access-6zz6h\") pod \"must-gather-6ss7s\" (UID: \"5762e105-bdd0-49fd-a477-4dfc9b5ae71d\") " pod="openshift-must-gather-tg9wr/must-gather-6ss7s"
Sep 08 14:28:20.402215 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:20.402194    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5762e105-bdd0-49fd-a477-4dfc9b5ae71d-must-gather-output\") pod \"must-gather-6ss7s\" (UID: \"5762e105-bdd0-49fd-a477-4dfc9b5ae71d\") " pod="openshift-must-gather-tg9wr/must-gather-6ss7s"
Sep 08 14:28:20.414301 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:20.414274    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zz6h\" (UniqueName: \"kubernetes.io/projected/5762e105-bdd0-49fd-a477-4dfc9b5ae71d-kube-api-access-6zz6h\") pod \"must-gather-6ss7s\" (UID: \"5762e105-bdd0-49fd-a477-4dfc9b5ae71d\") " pod="openshift-must-gather-tg9wr/must-gather-6ss7s"
Sep 08 14:28:20.543151 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:20.543045    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tg9wr/must-gather-6ss7s"
Sep 08 14:28:20.815127 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:20.815085    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tg9wr/must-gather-6ss7s"]
Sep 08 14:28:20.817645 ip-10-0-128-147 kubenswrapper[2558]: W0908 14:28:20.817615    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5762e105_bdd0_49fd_a477_4dfc9b5ae71d.slice/crio-9127604d07f26fce7dcc02c3f73edc800153a6c0943cafe32bce22203246ae2c WatchSource:0}: Error finding container 9127604d07f26fce7dcc02c3f73edc800153a6c0943cafe32bce22203246ae2c: Status 404 returned error can't find the container with id 9127604d07f26fce7dcc02c3f73edc800153a6c0943cafe32bce22203246ae2c
Sep 08 14:28:20.819405 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:20.819388    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 14:28:20.831417 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:20.831389    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tg9wr/must-gather-6ss7s" event={"ID":"5762e105-bdd0-49fd-a477-4dfc9b5ae71d","Type":"ContainerStarted","Data":"9127604d07f26fce7dcc02c3f73edc800153a6c0943cafe32bce22203246ae2c"}
Sep 08 14:28:28.885391 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:28.885340    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tg9wr/must-gather-6ss7s" event={"ID":"5762e105-bdd0-49fd-a477-4dfc9b5ae71d","Type":"ContainerStarted","Data":"9422b09064db7360e009f44d23cdcccc07908929cd7d5a42d1da6d0226f0b8fe"}
Sep 08 14:28:28.885391 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:28.885406    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tg9wr/must-gather-6ss7s" event={"ID":"5762e105-bdd0-49fd-a477-4dfc9b5ae71d","Type":"ContainerStarted","Data":"8497d0b3989d4fb86c54bcc4a04b2fe657527948c33e5b02de9300475f92eb2a"}
Sep 08 14:28:28.900432 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:28.900366    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tg9wr/must-gather-6ss7s" podStartSLOduration=1.5864028019999998 podStartE2EDuration="8.900343732s" totalImagesPullingTime="7.31394093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:28:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:28:28.89892415 +0000 UTC m=+4372.350685090" watchObservedRunningTime="2026-09-08 14:28:28.900343732 +0000 UTC m=+4372.352104664"
Sep 08 14:28:47.029548 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:47.029511    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-66v2g_938161a3-9204-4b80-a381-eac9102eca91/authorino/0.log"
Sep 08 14:28:47.095570 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:47.095540    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-r4kmc_b8cc5ec5-fcc5-4d39-903f-95a3afe1f61b/kuadrant-console-plugin/0.log"
Sep 08 14:28:51.012877 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:51.012842    2558 generic.go:356] "Generic (PLEG): container finished" podID="5762e105-bdd0-49fd-a477-4dfc9b5ae71d" containerID="8497d0b3989d4fb86c54bcc4a04b2fe657527948c33e5b02de9300475f92eb2a" exitCode=0
Sep 08 14:28:51.013449 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:51.012918    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tg9wr/must-gather-6ss7s" event={"ID":"5762e105-bdd0-49fd-a477-4dfc9b5ae71d","Type":"ContainerDied","Data":"8497d0b3989d4fb86c54bcc4a04b2fe657527948c33e5b02de9300475f92eb2a"}
Sep 08 14:28:51.013449 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:51.013341    2558 scope.go:117] "RemoveContainer" containerID="8497d0b3989d4fb86c54bcc4a04b2fe657527948c33e5b02de9300475f92eb2a"
Sep 08 14:28:51.121340 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:51.121309    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-tg9wr_must-gather-6ss7s_5762e105-bdd0-49fd-a477-4dfc9b5ae71d/gather/0.log"
Sep 08 14:28:51.779196 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:51.779155    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nm4b8/must-gather-kd7n2"]
Sep 08 14:28:51.783720 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:51.783700    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nm4b8/must-gather-kd7n2"
Sep 08 14:28:51.784340 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:51.784313    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0a425360-6c83-44bb-bca5-1b8f14e1060a-must-gather-output\") pod \"must-gather-kd7n2\" (UID: \"0a425360-6c83-44bb-bca5-1b8f14e1060a\") " pod="openshift-must-gather-nm4b8/must-gather-kd7n2"
Sep 08 14:28:51.784424 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:51.784353    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtwvq\" (UniqueName: \"kubernetes.io/projected/0a425360-6c83-44bb-bca5-1b8f14e1060a-kube-api-access-mtwvq\") pod \"must-gather-kd7n2\" (UID: \"0a425360-6c83-44bb-bca5-1b8f14e1060a\") " pod="openshift-must-gather-nm4b8/must-gather-kd7n2"
Sep 08 14:28:51.786396 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:51.786364    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nm4b8\"/\"default-dockercfg-fsrkr\""
Sep 08 14:28:51.786497 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:51.786413    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nm4b8\"/\"kube-root-ca.crt\""
Sep 08 14:28:51.787139 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:51.787121    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nm4b8\"/\"openshift-service-ca.crt\""
Sep 08 14:28:51.799130 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:51.799105    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nm4b8/must-gather-kd7n2"]
Sep 08 14:28:51.884938 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:51.884902    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtwvq\" (UniqueName: \"kubernetes.io/projected/0a425360-6c83-44bb-bca5-1b8f14e1060a-kube-api-access-mtwvq\") pod \"must-gather-kd7n2\" (UID: \"0a425360-6c83-44bb-bca5-1b8f14e1060a\") " pod="openshift-must-gather-nm4b8/must-gather-kd7n2"
Sep 08 14:28:51.885141 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:51.884982    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0a425360-6c83-44bb-bca5-1b8f14e1060a-must-gather-output\") pod \"must-gather-kd7n2\" (UID: \"0a425360-6c83-44bb-bca5-1b8f14e1060a\") " pod="openshift-must-gather-nm4b8/must-gather-kd7n2"
Sep 08 14:28:51.885267 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:51.885248    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0a425360-6c83-44bb-bca5-1b8f14e1060a-must-gather-output\") pod \"must-gather-kd7n2\" (UID: \"0a425360-6c83-44bb-bca5-1b8f14e1060a\") " pod="openshift-must-gather-nm4b8/must-gather-kd7n2"
Sep 08 14:28:51.900084 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:51.900060    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtwvq\" (UniqueName: \"kubernetes.io/projected/0a425360-6c83-44bb-bca5-1b8f14e1060a-kube-api-access-mtwvq\") pod \"must-gather-kd7n2\" (UID: \"0a425360-6c83-44bb-bca5-1b8f14e1060a\") " pod="openshift-must-gather-nm4b8/must-gather-kd7n2"
Sep 08 14:28:52.093012 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:52.092923    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nm4b8/must-gather-kd7n2"
Sep 08 14:28:52.393197 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:52.392915    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nm4b8/must-gather-kd7n2"]
Sep 08 14:28:52.395820 ip-10-0-128-147 kubenswrapper[2558]: W0908 14:28:52.395792    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a425360_6c83_44bb_bca5_1b8f14e1060a.slice/crio-00422afde31a8898c9b0ce74d9efccde186cf97bb6fb3bb6da28c72321739ab5 WatchSource:0}: Error finding container 00422afde31a8898c9b0ce74d9efccde186cf97bb6fb3bb6da28c72321739ab5: Status 404 returned error can't find the container with id 00422afde31a8898c9b0ce74d9efccde186cf97bb6fb3bb6da28c72321739ab5
Sep 08 14:28:53.026733 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:53.026691    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nm4b8/must-gather-kd7n2" event={"ID":"0a425360-6c83-44bb-bca5-1b8f14e1060a","Type":"ContainerStarted","Data":"00422afde31a8898c9b0ce74d9efccde186cf97bb6fb3bb6da28c72321739ab5"}
Sep 08 14:28:54.034755 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:54.034712    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nm4b8/must-gather-kd7n2" event={"ID":"0a425360-6c83-44bb-bca5-1b8f14e1060a","Type":"ContainerStarted","Data":"6d7894cf03c6c5cb0c0e9ff21cd204dee289675a5ea96312a84f2f16d8415019"}
Sep 08 14:28:54.035319 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:54.035295    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nm4b8/must-gather-kd7n2" event={"ID":"0a425360-6c83-44bb-bca5-1b8f14e1060a","Type":"ContainerStarted","Data":"68fdd70b8cab0c80e97b3b3e8cd90f4fa5b79ee7ae924eafee96015c03496787"}
Sep 08 14:28:54.054139 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:54.054073    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nm4b8/must-gather-kd7n2" podStartSLOduration=2.190599282 podStartE2EDuration="3.054050712s" totalImagesPullingTime="863.45143ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:28:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:28:54.052510149 +0000 UTC m=+4397.504271093" watchObservedRunningTime="2026-09-08 14:28:54.054050712 +0000 UTC m=+4397.505811645"
Sep 08 14:28:55.031830 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:55.031797    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-g5jt7_24a4cf01-77e6-44a0-8074-0929ee8dfee7/global-pull-secret-syncer/0.log"
Sep 08 14:28:55.247288 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:55.247258    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-k9f94_58dbf292-5174-4712-b380-d5763868b1c5/konnectivity-agent/0.log"
Sep 08 14:28:55.338450 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:55.338370    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-147.ec2.internal_21e691d07f9ad0e45bf40e33342a7334/haproxy/0.log"
Sep 08 14:28:56.619182 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:56.619143    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-tg9wr/must-gather-6ss7s"]
Sep 08 14:28:56.620697 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:56.620653    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-tg9wr/must-gather-6ss7s" podUID="5762e105-bdd0-49fd-a477-4dfc9b5ae71d" containerName="copy" containerID="cri-o://9422b09064db7360e009f44d23cdcccc07908929cd7d5a42d1da6d0226f0b8fe" gracePeriod=2
Sep 08 14:28:56.628505 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:56.628344    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-tg9wr/must-gather-6ss7s"]
Sep 08 14:28:57.000096 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:56.999574    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-tg9wr_must-gather-6ss7s_5762e105-bdd0-49fd-a477-4dfc9b5ae71d/copy/0.log"
Sep 08 14:28:57.003269 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:57.002685    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tg9wr/must-gather-6ss7s"
Sep 08 14:28:57.008620 ip-10-0-128-147 kubenswrapper[2558]: E0908 14:28:57.007588    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-6ss7s\" is forbidden: User \"system:node:ip-10-0-128-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-tg9wr\": no relationship found between node 'ip-10-0-128-147.ec2.internal' and this object" podUID="5762e105-bdd0-49fd-a477-4dfc9b5ae71d" pod="openshift-must-gather-tg9wr/must-gather-6ss7s"
Sep 08 14:28:57.029627 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:57.029365    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5762e105-bdd0-49fd-a477-4dfc9b5ae71d-must-gather-output\") pod \"5762e105-bdd0-49fd-a477-4dfc9b5ae71d\" (UID: \"5762e105-bdd0-49fd-a477-4dfc9b5ae71d\") "
Sep 08 14:28:57.029627 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:57.029485    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zz6h\" (UniqueName: \"kubernetes.io/projected/5762e105-bdd0-49fd-a477-4dfc9b5ae71d-kube-api-access-6zz6h\") pod \"5762e105-bdd0-49fd-a477-4dfc9b5ae71d\" (UID: \"5762e105-bdd0-49fd-a477-4dfc9b5ae71d\") "
Sep 08 14:28:57.035365 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:57.035324    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5762e105-bdd0-49fd-a477-4dfc9b5ae71d-kube-api-access-6zz6h" (OuterVolumeSpecName: "kube-api-access-6zz6h") pod "5762e105-bdd0-49fd-a477-4dfc9b5ae71d" (UID: "5762e105-bdd0-49fd-a477-4dfc9b5ae71d"). InnerVolumeSpecName "kube-api-access-6zz6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:28:57.043562 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:57.043508    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5762e105-bdd0-49fd-a477-4dfc9b5ae71d-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "5762e105-bdd0-49fd-a477-4dfc9b5ae71d" (UID: "5762e105-bdd0-49fd-a477-4dfc9b5ae71d"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:28:57.083619 ip-10-0-128-147 kubenswrapper[2558]: E0908 14:28:57.082882    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-6ss7s\" is forbidden: User \"system:node:ip-10-0-128-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-tg9wr\": no relationship found between node 'ip-10-0-128-147.ec2.internal' and this object" podUID="5762e105-bdd0-49fd-a477-4dfc9b5ae71d" pod="openshift-must-gather-tg9wr/must-gather-6ss7s"
Sep 08 14:28:57.084255 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:57.084225    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5762e105-bdd0-49fd-a477-4dfc9b5ae71d" path="/var/lib/kubelet/pods/5762e105-bdd0-49fd-a477-4dfc9b5ae71d/volumes"
Sep 08 14:28:57.092066 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:57.092037    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-tg9wr_must-gather-6ss7s_5762e105-bdd0-49fd-a477-4dfc9b5ae71d/copy/0.log"
Sep 08 14:28:57.092522 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:57.092495    2558 generic.go:356] "Generic (PLEG): container finished" podID="5762e105-bdd0-49fd-a477-4dfc9b5ae71d" containerID="9422b09064db7360e009f44d23cdcccc07908929cd7d5a42d1da6d0226f0b8fe" exitCode=143
Sep 08 14:28:57.092657 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:57.092565    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tg9wr/must-gather-6ss7s"
Sep 08 14:28:57.092657 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:57.092584    2558 scope.go:117] "RemoveContainer" containerID="9422b09064db7360e009f44d23cdcccc07908929cd7d5a42d1da6d0226f0b8fe"
Sep 08 14:28:57.121399 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:57.121375    2558 scope.go:117] "RemoveContainer" containerID="8497d0b3989d4fb86c54bcc4a04b2fe657527948c33e5b02de9300475f92eb2a"
Sep 08 14:28:57.130700 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:57.130660    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zz6h\" (UniqueName: \"kubernetes.io/projected/5762e105-bdd0-49fd-a477-4dfc9b5ae71d-kube-api-access-6zz6h\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:28:57.130901 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:57.130886    2558 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5762e105-bdd0-49fd-a477-4dfc9b5ae71d-must-gather-output\") on node \"ip-10-0-128-147.ec2.internal\" DevicePath \"\""
Sep 08 14:28:57.162877 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:57.162850    2558 scope.go:117] "RemoveContainer" containerID="9422b09064db7360e009f44d23cdcccc07908929cd7d5a42d1da6d0226f0b8fe"
Sep 08 14:28:57.163615 ip-10-0-128-147 kubenswrapper[2558]: E0908 14:28:57.163475    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9422b09064db7360e009f44d23cdcccc07908929cd7d5a42d1da6d0226f0b8fe\": container with ID starting with 9422b09064db7360e009f44d23cdcccc07908929cd7d5a42d1da6d0226f0b8fe not found: ID does not exist" containerID="9422b09064db7360e009f44d23cdcccc07908929cd7d5a42d1da6d0226f0b8fe"
Sep 08 14:28:57.163615 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:57.163516    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9422b09064db7360e009f44d23cdcccc07908929cd7d5a42d1da6d0226f0b8fe"} err="failed to get container status \"9422b09064db7360e009f44d23cdcccc07908929cd7d5a42d1da6d0226f0b8fe\": rpc error: code = NotFound desc = could not find container \"9422b09064db7360e009f44d23cdcccc07908929cd7d5a42d1da6d0226f0b8fe\": container with ID starting with 9422b09064db7360e009f44d23cdcccc07908929cd7d5a42d1da6d0226f0b8fe not found: ID does not exist"
Sep 08 14:28:57.163615 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:57.163540    2558 scope.go:117] "RemoveContainer" containerID="8497d0b3989d4fb86c54bcc4a04b2fe657527948c33e5b02de9300475f92eb2a"
Sep 08 14:28:57.164586 ip-10-0-128-147 kubenswrapper[2558]: E0908 14:28:57.164523    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8497d0b3989d4fb86c54bcc4a04b2fe657527948c33e5b02de9300475f92eb2a\": container with ID starting with 8497d0b3989d4fb86c54bcc4a04b2fe657527948c33e5b02de9300475f92eb2a not found: ID does not exist" containerID="8497d0b3989d4fb86c54bcc4a04b2fe657527948c33e5b02de9300475f92eb2a"
Sep 08 14:28:57.164586 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:28:57.164556    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8497d0b3989d4fb86c54bcc4a04b2fe657527948c33e5b02de9300475f92eb2a"} err="failed to get container status \"8497d0b3989d4fb86c54bcc4a04b2fe657527948c33e5b02de9300475f92eb2a\": rpc error: code = NotFound desc = could not find container \"8497d0b3989d4fb86c54bcc4a04b2fe657527948c33e5b02de9300475f92eb2a\": container with ID starting with 8497d0b3989d4fb86c54bcc4a04b2fe657527948c33e5b02de9300475f92eb2a not found: ID does not exist"
Sep 08 14:29:00.825743 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:00.825641    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cf4765b85-66v2g_938161a3-9204-4b80-a381-eac9102eca91/authorino/0.log"
Sep 08 14:29:01.111470 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:01.111432    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-r4kmc_b8cc5ec5-fcc5-4d39-903f-95a3afe1f61b/kuadrant-console-plugin/0.log"
Sep 08 14:29:02.859717 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:02.859691    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ab57a63e-5ff7-4bdb-a5b9-70947b872230/alertmanager/0.log"
Sep 08 14:29:02.990859 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:02.990819    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ab57a63e-5ff7-4bdb-a5b9-70947b872230/config-reloader/0.log"
Sep 08 14:29:03.068086 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:03.068032    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ab57a63e-5ff7-4bdb-a5b9-70947b872230/kube-rbac-proxy-web/0.log"
Sep 08 14:29:03.129000 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:03.128911    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ab57a63e-5ff7-4bdb-a5b9-70947b872230/kube-rbac-proxy/0.log"
Sep 08 14:29:03.171081 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:03.171046    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ab57a63e-5ff7-4bdb-a5b9-70947b872230/kube-rbac-proxy-metric/0.log"
Sep 08 14:29:03.216705 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:03.216659    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ab57a63e-5ff7-4bdb-a5b9-70947b872230/prom-label-proxy/0.log"
Sep 08 14:29:03.256411 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:03.256381    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ab57a63e-5ff7-4bdb-a5b9-70947b872230/init-config-reloader/0.log"
Sep 08 14:29:03.567803 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:03.567703    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-jxhsg_0686fffb-a8a9-459f-878e-e434089eaf3d/monitoring-plugin/0.log"
Sep 08 14:29:03.744104 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:03.744061    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p84l2_e84ecd52-fe2b-4159-99e1-dd97713d9a65/node-exporter/0.log"
Sep 08 14:29:03.785952 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:03.785919    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p84l2_e84ecd52-fe2b-4159-99e1-dd97713d9a65/kube-rbac-proxy/0.log"
Sep 08 14:29:03.830754 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:03.830673    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p84l2_e84ecd52-fe2b-4159-99e1-dd97713d9a65/init-textfile/0.log"
Sep 08 14:29:04.521543 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:04.521514    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-x6c56_09b7c4f8-08f1-4730-93cb-e9672f1e09d3/prometheus-operator-admission-webhook/0.log"
Sep 08 14:29:04.565251 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:04.565220    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-787d46f89f-pgss5_f553b081-188a-47fa-9cac-a798cef2d48f/telemeter-client/0.log"
Sep 08 14:29:04.610568 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:04.610542    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-787d46f89f-pgss5_f553b081-188a-47fa-9cac-a798cef2d48f/reload/0.log"
Sep 08 14:29:04.643926 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:04.643887    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-787d46f89f-pgss5_f553b081-188a-47fa-9cac-a798cef2d48f/kube-rbac-proxy/0.log"
Sep 08 14:29:04.694308 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:04.694282    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-cd4f7799f-bnz9h_cc05686f-7483-4834-afcc-f89ce4721ccc/thanos-query/0.log"
Sep 08 14:29:04.730497 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:04.730469    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-cd4f7799f-bnz9h_cc05686f-7483-4834-afcc-f89ce4721ccc/kube-rbac-proxy-web/0.log"
Sep 08 14:29:04.786855 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:04.786760    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-cd4f7799f-bnz9h_cc05686f-7483-4834-afcc-f89ce4721ccc/kube-rbac-proxy/0.log"
Sep 08 14:29:04.870159 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:04.870125    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nm4b8/perf-node-gather-daemonset-c8lgd"]
Sep 08 14:29:04.870519 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:04.870506    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5762e105-bdd0-49fd-a477-4dfc9b5ae71d" containerName="gather"
Sep 08 14:29:04.870570 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:04.870522    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5762e105-bdd0-49fd-a477-4dfc9b5ae71d" containerName="gather"
Sep 08 14:29:04.870570 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:04.870544    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5762e105-bdd0-49fd-a477-4dfc9b5ae71d" containerName="copy"
Sep 08 14:29:04.870570 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:04.870549    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5762e105-bdd0-49fd-a477-4dfc9b5ae71d" containerName="copy"
Sep 08 14:29:04.870698 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:04.870630    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5762e105-bdd0-49fd-a477-4dfc9b5ae71d" containerName="gather"
Sep 08 14:29:04.870698 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:04.870641    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5762e105-bdd0-49fd-a477-4dfc9b5ae71d" containerName="copy"
Sep 08 14:29:04.875571 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:04.875541    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-c8lgd"
Sep 08 14:29:04.881783 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:04.881760    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-cd4f7799f-bnz9h_cc05686f-7483-4834-afcc-f89ce4721ccc/prom-label-proxy/0.log"
Sep 08 14:29:04.905612 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:04.905564    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nm4b8/perf-node-gather-daemonset-c8lgd"]
Sep 08 14:29:04.912195 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:04.912164    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4c90d251-c268-465f-9e1f-6084c55c2729-podres\") pod \"perf-node-gather-daemonset-c8lgd\" (UID: \"4c90d251-c268-465f-9e1f-6084c55c2729\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-c8lgd"
Sep 08 14:29:04.912335 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:04.912277    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c90d251-c268-465f-9e1f-6084c55c2729-sys\") pod \"perf-node-gather-daemonset-c8lgd\" (UID: \"4c90d251-c268-465f-9e1f-6084c55c2729\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-c8lgd"
Sep 08 14:29:04.912335 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:04.912317    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpvmp\" (UniqueName: \"kubernetes.io/projected/4c90d251-c268-465f-9e1f-6084c55c2729-kube-api-access-tpvmp\") pod \"perf-node-gather-daemonset-c8lgd\" (UID: \"4c90d251-c268-465f-9e1f-6084c55c2729\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-c8lgd"
Sep 08 14:29:04.912442 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:04.912373    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4c90d251-c268-465f-9e1f-6084c55c2729-proc\") pod \"perf-node-gather-daemonset-c8lgd\" (UID: \"4c90d251-c268-465f-9e1f-6084c55c2729\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-c8lgd"
Sep 08 14:29:04.912442 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:04.912397    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c90d251-c268-465f-9e1f-6084c55c2729-lib-modules\") pod \"perf-node-gather-daemonset-c8lgd\" (UID: \"4c90d251-c268-465f-9e1f-6084c55c2729\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-c8lgd"
Sep 08 14:29:04.975811 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:04.975759    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-cd4f7799f-bnz9h_cc05686f-7483-4834-afcc-f89ce4721ccc/kube-rbac-proxy-rules/0.log"
Sep 08 14:29:05.013440 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:05.013406    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4c90d251-c268-465f-9e1f-6084c55c2729-podres\") pod \"perf-node-gather-daemonset-c8lgd\" (UID: \"4c90d251-c268-465f-9e1f-6084c55c2729\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-c8lgd"
Sep 08 14:29:05.013649 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:05.013492    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c90d251-c268-465f-9e1f-6084c55c2729-sys\") pod \"perf-node-gather-daemonset-c8lgd\" (UID: \"4c90d251-c268-465f-9e1f-6084c55c2729\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-c8lgd"
Sep 08 14:29:05.013649 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:05.013520    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpvmp\" (UniqueName: \"kubernetes.io/projected/4c90d251-c268-465f-9e1f-6084c55c2729-kube-api-access-tpvmp\") pod \"perf-node-gather-daemonset-c8lgd\" (UID: \"4c90d251-c268-465f-9e1f-6084c55c2729\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-c8lgd"
Sep 08 14:29:05.013649 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:05.013564    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4c90d251-c268-465f-9e1f-6084c55c2729-podres\") pod \"perf-node-gather-daemonset-c8lgd\" (UID: \"4c90d251-c268-465f-9e1f-6084c55c2729\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-c8lgd"
Sep 08 14:29:05.013649 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:05.013571    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4c90d251-c268-465f-9e1f-6084c55c2729-proc\") pod \"perf-node-gather-daemonset-c8lgd\" (UID: \"4c90d251-c268-465f-9e1f-6084c55c2729\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-c8lgd"
Sep 08 14:29:05.013649 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:05.013624    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c90d251-c268-465f-9e1f-6084c55c2729-lib-modules\") pod \"perf-node-gather-daemonset-c8lgd\" (UID: \"4c90d251-c268-465f-9e1f-6084c55c2729\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-c8lgd"
Sep 08 14:29:05.013649 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:05.013640    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4c90d251-c268-465f-9e1f-6084c55c2729-proc\") pod \"perf-node-gather-daemonset-c8lgd\" (UID: \"4c90d251-c268-465f-9e1f-6084c55c2729\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-c8lgd"
Sep 08 14:29:05.013649 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:05.013641    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c90d251-c268-465f-9e1f-6084c55c2729-sys\") pod \"perf-node-gather-daemonset-c8lgd\" (UID: \"4c90d251-c268-465f-9e1f-6084c55c2729\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-c8lgd"
Sep 08 14:29:05.013991 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:05.013754    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c90d251-c268-465f-9e1f-6084c55c2729-lib-modules\") pod \"perf-node-gather-daemonset-c8lgd\" (UID: \"4c90d251-c268-465f-9e1f-6084c55c2729\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-c8lgd"
Sep 08 14:29:05.027885 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:05.027854    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpvmp\" (UniqueName: \"kubernetes.io/projected/4c90d251-c268-465f-9e1f-6084c55c2729-kube-api-access-tpvmp\") pod \"perf-node-gather-daemonset-c8lgd\" (UID: \"4c90d251-c268-465f-9e1f-6084c55c2729\") " pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-c8lgd"
Sep 08 14:29:05.033691 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:05.033660    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-cd4f7799f-bnz9h_cc05686f-7483-4834-afcc-f89ce4721ccc/kube-rbac-proxy-metrics/0.log"
Sep 08 14:29:05.185407 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:05.185371    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-c8lgd"
Sep 08 14:29:05.482911 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:05.482826    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nm4b8/perf-node-gather-daemonset-c8lgd"]
Sep 08 14:29:05.487272 ip-10-0-128-147 kubenswrapper[2558]: W0908 14:29:05.487235    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4c90d251_c268_465f_9e1f_6084c55c2729.slice/crio-cf480ac73db4980fb3a045812897b7fee01f3ca7345a57a24ceaefa50c89c877 WatchSource:0}: Error finding container cf480ac73db4980fb3a045812897b7fee01f3ca7345a57a24ceaefa50c89c877: Status 404 returned error can't find the container with id cf480ac73db4980fb3a045812897b7fee01f3ca7345a57a24ceaefa50c89c877
Sep 08 14:29:06.170855 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:06.170812    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-c8lgd" event={"ID":"4c90d251-c268-465f-9e1f-6084c55c2729","Type":"ContainerStarted","Data":"888cde771d36033f35810ef1643d015de9246ed829e017046e71181831045806"}
Sep 08 14:29:06.171281 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:06.170883    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-c8lgd" event={"ID":"4c90d251-c268-465f-9e1f-6084c55c2729","Type":"ContainerStarted","Data":"cf480ac73db4980fb3a045812897b7fee01f3ca7345a57a24ceaefa50c89c877"}
Sep 08 14:29:06.171281 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:06.170994    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-c8lgd"
Sep 08 14:29:07.443154 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:07.443119    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-hz955_46b98f99-3985-4376-818e-269810605864/download-server/0.log"
Sep 08 14:29:09.365637 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:09.365588    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pvnnp_7bcc0442-53c5-4088-a419-10e4af66a483/dns/0.log"
Sep 08 14:29:09.391943 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:09.391909    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pvnnp_7bcc0442-53c5-4088-a419-10e4af66a483/kube-rbac-proxy/0.log"
Sep 08 14:29:09.483033 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:09.483001    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vwvwg_3ce2ec68-0166-484d-8e1a-28cc99334e5d/dns-node-resolver/0.log"
Sep 08 14:29:10.116479 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:10.116451    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7t44n_eb60fb08-544c-4bd8-93fe-103f5f2bde69/node-ca/0.log"
Sep 08 14:29:11.982170 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:11.982142    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-d2mvr_349aa96b-eaec-45cf-9a20-7f2227273c2c/serve-healthcheck-canary/0.log"
Sep 08 14:29:12.188895 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:12.188864    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-c8lgd"
Sep 08 14:29:12.208608 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:12.208531    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nm4b8/perf-node-gather-daemonset-c8lgd" podStartSLOduration=8.208507915 podStartE2EDuration="8.208507915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 14:29:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:29:06.193877018 +0000 UTC m=+4409.645637939" watchObservedRunningTime="2026-09-08 14:29:12.208507915 +0000 UTC m=+4415.660268847"
Sep 08 14:29:12.519990 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:12.519951    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-4q6fg_e818a776-d654-41ac-ba74-e47a0532f4d6/insights-operator/0.log"
Sep 08 14:29:12.521415 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:12.521387    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-4q6fg_e818a776-d654-41ac-ba74-e47a0532f4d6/insights-operator/1.log"
Sep 08 14:29:12.549089 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:12.549057    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8nbzm_5872b485-7dfa-45e1-8313-2393fc7361e6/kube-rbac-proxy/0.log"
Sep 08 14:29:12.579701 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:12.579669    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8nbzm_5872b485-7dfa-45e1-8313-2393fc7361e6/exporter/0.log"
Sep 08 14:29:12.605452 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:12.605424    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8nbzm_5872b485-7dfa-45e1-8313-2393fc7361e6/extractor/0.log"
Sep 08 14:29:16.658246 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:16.658212    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-66c568d8-t6qln_2d213d1c-5fbd-4f74-881d-9754ee40eb1a/manager/0.log"
Sep 08 14:29:16.706815 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:16.706768    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-kzjrd_0fa216db-06cc-4816-8010-a7bb50bb3a88/openshift-lws-operator/0.log"
Sep 08 14:29:18.928763 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:18.928721    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-vcwmq_acb16799-7937-4c56-824b-fb8cc954cc22/manager/0.log"
Sep 08 14:29:19.020201 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:19.020163    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-dnq2z_ef854386-230c-47e6-9e11-e538b12a9e68/s3-init/0.log"
Sep 08 14:29:19.059677 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:19.059649    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-dnq2z_ef854386-230c-47e6-9e11-e538b12a9e68/download-hf-model/0.log"
Sep 08 14:29:19.113312 ip-10-0-128-147 kubenswrapper[2558]: I0908 14:29:19.113257    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-bql4r_dfcdf8b7-003b-470f-bd21-19f5db5b0a4c/seaweedfs/0.log"